Trilltechnician/giglez
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

test: add real-world Flipper Zero capture benchmark - CRITICAL FINDINGS

Browse Source 
## Summary

Added real-world testing infrastructure with 11 actual Flipper Zero community captures. Results reveal **CRITICAL GAP**: 0% accuracy on real captures vs 33% on synthetic signals.

## What Was Added

### Test Infrastructure
- **tests/real_captures/** - 11 real .sub files from UberGuidoZ/Flipper repository
  - 3 weather sensors (LaCrosse, Acurite, Nexus)
  - 2 garage doors (LiftMaster 433MHz, Security+ 2.0)
  - 2 doorbells (GE, Byron)
  - 1 LED remote
  - 3 ceiling fan controls

- **tests/real_captures/manifest.json** - Test case metadata with expected protocols

- **scripts/benchmark_real.py** - Real-world benchmark runner
  - Loads manifest.json
  - Runs identification on each capture
  - Generates REAL_TEST_RESULTS.md
  - Compares with synthetic results

### Documentation
- **REAL_TEST_RESULTS.md** - Benchmark results (0% accuracy)
- **docs/REAL_CAPTURE_ANALYSIS.md** - Comprehensive gap analysis

## Results

### Performance Comparison

| Metric | Synthetic | Real-World | Delta |
|--------|-----------|------------|-------|
| Top-1 Accuracy | 33.3% | **0.0%** | **-33.3%** |
| Top-3 Accuracy | 50.0% | **0.0%** | **-50.0%** |
| High Confidence | 66.7% | 9.1% | -57.6% |
| No Matches | 8.3% | **45.5%** | +37.2% |

### Root Causes Identified

1. **Decoded File Format (45% of failures)** - CRITICAL
   - 5/11 files are already decoded (Protocol: Holtek_HT12X), not RAW
   - Our system ONLY processes RAW_Data
   - All ceiling fan and LED remote files fail immediately
   - **Fix**: Add support for parsed decode .sub file format

2. **Missing Protocols (27% of failures)**
   - Acurite 02077M not in database (we have 609TXC)
   - GE Doorbell 19297 not in database
   - Byron DB421E not in database
   - LiftMaster not in database
   - **Fix**: Add these protocols from real captures

3. **Real Signal Complexity (27% of failures)**
   - LaCrosse real capture: 131 pulses (multi-packet)
   - Synthetic LaCrosse: 40 bits (single packet)
   - Real signals have noise, jitter, interference
   - **Fix**: Packet segmentation, higher noise tolerance

4. **Protocol Family Competition (9% of failures)**
   - Nexus found at rank 34 (beaten by Oregon Scientific)
   - Similar protocols competing instead of grouping
   - **Fix**: Protocol family scoring

## Key Insights

### Why Synthetic Worked (33% accuracy):
- Perfect timing with controlled 5-15% jitter
- Single packet per file
- All RAW format
- Known protocol parameters

### Why Real Failed (0% accuracy):
- 45% already decoded (not RAW)
- 27% missing from database
- Variable signal quality
- Multi-packet transmissions
- Real-world interference

**Conclusion**: System was optimized for unrealistic synthetic signals.

## Recommendations (Priority Order)

1. **P1 - Support Decoded Files** (+45% accuracy)
   - Parse Protocol, Bit, Key, TE fields
   - Match by protocol name + timing element
   - Skip RAW analysis for decoded files

2. **P2 - Add Missing Protocols** (+27% accuracy)
   - Import Acurite 02077M, GE/Byron doorbells, LiftMaster, Holtek HT12X
   - Source from real captures or RTL_433 updates

3. **P3 - Improve Robustness** (+18% accuracy)
   - Packet segmentation for multi-transmission captures
   - Increase jitter tolerance to 25%
   - Repetition detection

4. **P4 - Family Grouping** (+9% accuracy)
   - Group similar protocols (Nexus/Oregon/Acurite)
   - Boost exact name matches

**Expected Final Accuracy**: 60-70% after P1+P2 implementation.

## Files Changed

- tests/real_captures/*.sub (11 files, 192KB)
- tests/real_captures/manifest.json (11 test cases)
- scripts/benchmark_real.py (370 lines)
- REAL_TEST_RESULTS.md (benchmark results)
- docs/REAL_CAPTURE_ANALYSIS.md (comprehensive analysis)

🎯 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
leetcrypt
2026-02-15 18:08:39 -08:00
parent 1db471822d
commit 6945ece853
15 changed files with 1126 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/real_captures/byron_doorbell_real.sub
+11
View File
@@ -0,0 +1,11 @@
Filetype: Flipper SubGhz RAW File
Version: 1
#
# Byron_DB421E Doorbell
#
Frequency: 433920000
Preset: FuriHalSubGhzPresetOok650Async
Protocol: RAW
RAW_Data: 342905 -7094 471 -1538 1473 -554 451 -1542 1471 -538 1469 -520 463 -1570 447 -1564 453 -1530 485 -1544 1475 -542 445 -1544 479 -1520 1467 -548 491 -1540 447 -1558 1447 -548 1475 -554 445 -1558 453 -1542 487 -1512 493 -1540 1461 -518 483 -1550 1467 -542 1467 -516 1483 -524 473 -1560 1457 -536 473 -1542 445 -1560 1481 -550 1457 -524 473 -7108 455 -1546 1469 -536 441 -1566 1479 -550 1467 -520 455 -1550 447 -1562 471 -1550 447 -1542 1503 -520 449 -1548 449 -1572 1469 -542 445 -1572 441 -1566 1481 -526 1459 -524 451 -1580 445 -1570 441 -1564 441 -1568 1475 -518 447 -1558 1473 -546 1447 -564 1449 -558 467 -1542 1473 -518 483 -1550 445 -1562 1461 -544 1443 -552 479 -7070 481 -1558 1445 -538 471 -1536 1491 -520 1461 -558 473 -1542 447 -1556 487 -1526 447 -1548 1479 -518 495 -1536 473 -1528 1467 -536 473 -1540 481 -1522 1471 -548 1445 -568 441 -1548 483 -1544 447 -1550 485 -1542 1439 -566 441 -1546 1471 -556 1475 -518 1477 -550 445 -1550 1481 -522 459 -1576 449 -1548 1469 -550 1465 -520 489 -7086 445 -1548 1463 -562 435 -1574 1475 -516 1475 -550 447 -1552 439 -1562 459 -1540 451 -1570 1473 -550 445 -1548 441 -1568 1459 -540 477 -1546 449 -1544 1467 -538 1499 -516 485 -1544 447 -1548 473 -1532 493 -1542 1439 -544 481 -1542 1465 -520 1471 -546 1483 -520 467 -1568 1459 -528 487 -1532 479 -1514 1471 -548 1481 -544 455 -7070 499 -1540 1455 -554 459 -1540 1465 -554 1447 -568 441 -1544 447 -1558 485 -1520 483 -1544 1449 -570 441 -1540 445 -1588 1431 -548 459 -1568 441 -1578 1441 -550 1475 -536 439 -1564 461 -1538 471 -1560 449 -1544 1477 -534 473 -1540 1467 -556 1473 -542 1447 -550 447 -1550 1487 -530 471 -1540 447 -1564 1443 -584 1423 -588 429 -7094 493 -1542 1469 -518 483 -1544 1473 -522 1475 -546 453 -1544 487 -1540 463 -1538 447 -1562 1469 -546 455 -1536 471 -1528 1471 -542 483 -1542 457 -1540 1467 -556 1473 -544 449 -1540 449 -1572 441 -1544 477 -1538 1495 -520 449 -1546 1513 -496 1493 -544 1471 -516 477 -1554 1451 -550 455 -1568 449 -1528 1481 -546 1467 -556 473 -7060 479 -1544 1449 -556 467 -1544 1473 -516 1477 -546 477 -1534 487 -1514 491 -1510 485 -1548 1441 -554 479 -1520 481 -1528 1475 -536 487 -1540 453 -1538 1481 -530 1467 -556 447 -1560 451 -1544 477 -1530 471 -1550 1477 -518 481 -1518 1501 -520 1479 -538 1499 -520 451 -1544 1471 -534 471 -1542 473 -1538 1467 -554 1471 -544 451 -7092 457 -1538 1505 -520 445 -1562 1477 -514 1475 -560 487 -1504 481 -1548 487 -1506 489 -1536 1473 -514 483 -1546 477 -1508 1493 -510 511 -1514 479 -1518 1475 -546 1477 -536 471 -1524 471 -1546 475 -1538 479 -1520 1467 -548 455 -1570 1471 -518
RAW_Data: 1483 -516 1503 -506 487 -1530 1475 -534 487 -1538 451 -1536 1495 -518 1481 -548 447 -7078 491 -1530 1473 -534 471 -1536 1489 -526 1483 -544 445 -1534 475 -1550 473 -1528 461 -1566 1463 -548 449 -1548 479 -1510 1489 -536 471 -1544 481 -1522 1475 -546 1449 -536 501 -1502 497 -1542 449 -1526 485 -1552 1467 -524 481 -1524 1477 -536 1485 -522 1491 -526 489 -1516 1489 -526 455 -1548 473 -1538 1495 -504 1501 -522 491 -7048 505 -1522 1473 -550 447 -1562 1461 -544 1471 -548 451 -1536 441 -1582 445 -1542 479 -1554 1449 -550 459 -1534 471 -1558 1463 -536 471 -1542 479 -1520 1469 -548 1475 -538 475 -1514 481 -1528 483 -1538 471 -1526 1475 -546 477 -1528 1463 -548 1477 -516 1471 -556 487 -1524 1473 -522 481 -1556 453 -1542 1483 -542 1471 -518 479 -7084 455 -1548 1471 -538 443 -1578 1471 -518 1499 -506 485 -1558 451 -1542 457 -1542 473 -1540 1475 -542 449 -1546 449 -1578 1449 -562 429 -1572 447 -1566 1449 -548 1471 -540 471 -1530 489 -1542 449 -1530 485 -1550 1441 -554 481 -1520 1471 -538 1457 -562 1463 -554 473 -1528 1467 -534 473 -1536 475 -1548 1463 -552 1453 -538 475 -7070 479 -1530 1469 -536 471 -1554 1481 -514 1475 -532 467 -1572 465 -1538 449 -1546 483 -1548 1473 -518 449 -1550 481 -1532 1503 -512 481 -1546 443 -1560 1463 -544 1469 -546 441 -1568 449 -1544 461 -1564 439 -1560 1463 -536 473 -1548 1473 -546 1449 -546 1471 -556 443 -1560 1441 -568 441 -1556 485 -1526 1473 -522 1473 -548 461 -7100 485 -1518 1483 -528 471 -1546 1439 -552 1463 -554 453 -1558 447 -1580 429 -1572 465 -1538 1473 -518 485 -1542 449 -1546 1483 -524 461 -1572 449 -1548 1475 -548 1469 -520 449 -1550 483 -1542 457 -1540 473 -1542 1467 -554 453 -1558 1471 -520 1473 -546 1475 -538 471 -1530 1459 -536 509 -1508 475 -1550 1447 -548 1451 -556 487 -7068 481 -1548 1451 -536 475 -1544 1471 -548 1439 -578 449 -1542 485 -1512 493 -1542 449 -1546 1475 -518 479 -1546 473 -1540 1461 -524 483 -1546 449 -1542 1503 -520 1473 -550 449 -1528 473 -1558 449 -1544 483 -1538 1465 -546 443 -1544 1505 -518 1475 -546 1447 -564 473 -1528 1471 -550 445 -1546 471 -1544 1467 -522 1499 -550 449 -7078 479 -1552 1467 -518 493 -1544 1443 -546 1475 -550 445 -1548 487 -1508 483 -1534 473 -1566 1445 -548 445 -1550 485 -1542 1443 -568 431 -1568 467 -1538 1467 -552 1473 -516 483 -1548 429 -1570 467 -1540 447 -1546 1481 -548 447 -1576 1461 -542 1473 -520 1469 -552 453 -1548 1469 -538 443 -1564 457 -1568 1439 -546 1471 -548 477 -7084 487 -1510 1481 -524 471 -1562 1449 -548 1477 -536 471 -1532 487 -1506 485 -1538 471 -1526 1477 -548 481 -1512 467 -1538 1489 -528 471 -1560 451 -1544 1469 -520
RAW_Data: 1471 -548 461 -1566 441 -1546 481 -1542 447 -1544 1483 -526 495 -1532 1463 -552 1473 -512 1507 -496 471 -1546 1505 -520 447 -1546 483 -1544 1467 -556 1451 -548 453 -7096 459 -1542 1495 -508 479 -1548 1465 -536 1457 -542 479 -1550 475 -1524 491 -1510 487 -1518 1497 -520 481 -1516 481 -1558 1447 -538 469 -1534 497 -1510 1475 -548 1477 -550 445 -1546 489 -1506 483 -1542 485 -1506 1491 -526 471 -1550 1475 -518 1465 -554 1483 -516 479 -1526 1495 -518 487 -1510 479 -1548 1491 -510 1505 -518 447 -7114 453 -1544 1471 -538 471 -1534 1473 -550 1465 -538 441 -1582 445 -1538 481 -1552 453 -1530 1475 -556 453 -1560 455 -1540 1469 -546 479 -1514 479 -1554 1465 -518 1483 -524 473 -1560 481 -1520 477 -1530 471 -1526 1479 -548 447 -1542 1459 -540 1471 -550 1463 -554 489 -1524 1471 -522 481 -1526 483 -1542 1477 -510 1507 -518 481 -7080 455 -1556 1475 -520 487 -1554 1443 -532 1497 -542 441 -1538 473 -1562 441 -1536 475 -1538 1503 -520 449 -1550 449 -1572 1465 -546 445 -1574 447 -1548 1463 -536 1465 -554 445 -1560 473 -1534 469 -1542 447 -1554 1473 -546 453 -1566 1433 -546 1479 -546 1469 -554 443 -1562 1441 -538 475 -1560 473 -1532 1461 -558 1441 -548 483 -7070 479 -1544 1461 -536 477 -1542 1471 -518 1479 -546 479 -1512 499 -1532 467 -1540 445 -1558 1483 -512 483 -1536 471 -1548 1477 -518 479 -1520 481 -1524 1511 -496 1491 -544 479 -1540 439 -1562 439 -1548 475 -1538 1505 -520 449 -1548 1471 -536 1495 -508 1503 -520 445 -1556 1469 -548 455 -1566 449 -1560 1461 -534 1467 -155732 129 -364 163 -594 131 -300 67 -66 97 -398 327 -230 327 -526 65 -496 821 -66 855 -66 361 -132 8369 -9970 65 -1876 163 -624 65 -132 99 -532 231 -100 97 -166 97 -66 263 -494 329 -100 861 -100 329 -15258 129 -360 97 -166 97 -132 197 -196 97 -198 195 -468 165 -200 131 -166 97 -228 65 -164 859 -130 523 -164 195 -4622 67 -7968 165 -166 99 -66 395 -298 65 -492 65 -66 129 -428 65 -64 99 -1086 591 -132 195 -66 263 -98 163 -132 65