leetcrypt
6945ece853
## Summary Added real-world testing infrastructure with 11 actual Flipper Zero community captures. Results reveal **CRITICAL GAP**: 0% accuracy on real captures vs 33% on synthetic signals. ## What Was Added ### Test Infrastructure - **tests/real_captures/** - 11 real .sub files from UberGuidoZ/Flipper repository - 3 weather sensors (LaCrosse, Acurite, Nexus) - 2 garage doors (LiftMaster 433MHz, Security+ 2.0) - 2 doorbells (GE, Byron) - 1 LED remote - 3 ceiling fan controls - **tests/real_captures/manifest.json** - Test case metadata with expected protocols - **scripts/benchmark_real.py** - Real-world benchmark runner - Loads manifest.json - Runs identification on each capture - Generates REAL_TEST_RESULTS.md - Compares with synthetic results ### Documentation - **REAL_TEST_RESULTS.md** - Benchmark results (0% accuracy) - **docs/REAL_CAPTURE_ANALYSIS.md** - Comprehensive gap analysis ## Results ### Performance Comparison | Metric | Synthetic | Real-World | Delta | |--------|-----------|------------|-------| | Top-1 Accuracy | 33.3% | **0.0%** | **-33.3%** | | Top-3 Accuracy | 50.0% | **0.0%** | **-50.0%** | | High Confidence | 66.7% | 9.1% | -57.6% | | No Matches | 8.3% | **45.5%** | +37.2% | ### Root Causes Identified 1. **Decoded File Format (45% of failures)** - CRITICAL - 5/11 files are already decoded (Protocol: Holtek_HT12X), not RAW - Our system ONLY processes RAW_Data - All ceiling fan and LED remote files fail immediately - **Fix**: Add support for parsed decode .sub file format 2. **Missing Protocols (27% of failures)** - Acurite 02077M not in database (we have 609TXC) - GE Doorbell 19297 not in database - Byron DB421E not in database - LiftMaster not in database - **Fix**: Add these protocols from real captures 3. **Real Signal Complexity (27% of failures)** - LaCrosse real capture: 131 pulses (multi-packet) - Synthetic LaCrosse: 40 bits (single packet) - Real signals have noise, jitter, interference - **Fix**: Packet segmentation, higher noise tolerance 4. **Protocol Family Competition (9% of failures)** - Nexus found at rank 34 (beaten by Oregon Scientific) - Similar protocols competing instead of grouping - **Fix**: Protocol family scoring ## Key Insights ### Why Synthetic Worked (33% accuracy): - Perfect timing with controlled 5-15% jitter - Single packet per file - All RAW format - Known protocol parameters ### Why Real Failed (0% accuracy): - 45% already decoded (not RAW) - 27% missing from database - Variable signal quality - Multi-packet transmissions - Real-world interference **Conclusion**: System was optimized for unrealistic synthetic signals. ## Recommendations (Priority Order) 1. **P1 - Support Decoded Files** (+45% accuracy) - Parse Protocol, Bit, Key, TE fields - Match by protocol name + timing element - Skip RAW analysis for decoded files 2. **P2 - Add Missing Protocols** (+27% accuracy) - Import Acurite 02077M, GE/Byron doorbells, LiftMaster, Holtek HT12X - Source from real captures or RTL_433 updates 3. **P3 - Improve Robustness** (+18% accuracy) - Packet segmentation for multi-transmission captures - Increase jitter tolerance to 25% - Repetition detection 4. **P4 - Family Grouping** (+9% accuracy) - Group similar protocols (Nexus/Oregon/Acurite) - Boost exact name matches **Expected Final Accuracy**: 60-70% after P1+P2 implementation. ## Files Changed - tests/real_captures/*.sub (11 files, 192KB) - tests/real_captures/manifest.json (11 test cases) - scripts/benchmark_real.py (370 lines) - REAL_TEST_RESULTS.md (benchmark results) - docs/REAL_CAPTURE_ANALYSIS.md (comprehensive analysis) 🎯 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
96 lines
3.3 KiB
JSON
96 lines
3.3 KiB
JSON
{
|
|
"test_suite": "Real-World Flipper Zero Captures",
|
|
"source": "https://github.com/UberGuidoZ/Flipper",
|
|
"description": "Community-contributed .sub files from actual RF devices",
|
|
"test_cases": [
|
|
{
|
|
"filename": "lacrosse_tx141_real.sub",
|
|
"expected_protocol": "LaCrosse TX141-BV2",
|
|
"category": "weather_sensor",
|
|
"frequency": 433920000,
|
|
"source_path": "Weather_stations/LaCrosse-TX141TH-BV2/LaCrosse-TX141TH-BV2-raw.sub",
|
|
"notes": "Real weather station capture"
|
|
},
|
|
{
|
|
"filename": "acurite_02077m_real.sub",
|
|
"expected_protocol": "Acurite 02077M",
|
|
"category": "weather_sensor",
|
|
"frequency": 433920000,
|
|
"source_path": "Weather_stations/Acurite_02077M/RAW_2022.10.21-18.01.44.sub",
|
|
"notes": "Real Acurite weather station"
|
|
},
|
|
{
|
|
"filename": "nexus_th_real.sub",
|
|
"expected_protocol": "Nexus Temperature-Humidity",
|
|
"category": "weather_sensor",
|
|
"frequency": 433920000,
|
|
"source_path": "Weather_stations/Nexus-TH/nexus-th_raw.sub",
|
|
"notes": "Nexus temperature/humidity sensor"
|
|
},
|
|
{
|
|
"filename": "liftmaster_433_real.sub",
|
|
"expected_protocol": "LiftMaster",
|
|
"category": "garage_door",
|
|
"frequency": 433920000,
|
|
"source_path": "Garages/LiftMaster/LiftMaster_433MHz.sub",
|
|
"notes": "LiftMaster garage door opener"
|
|
},
|
|
{
|
|
"filename": "liftmaster_security2_raw_real.sub",
|
|
"expected_protocol": "LiftMaster Security+ 2.0",
|
|
"category": "garage_door",
|
|
"frequency": 315000000,
|
|
"source_path": "Garages/Security_2.0/7964/raw/893MAX_7964_raw.sub",
|
|
"notes": "LiftMaster Security+ 2.0 rolling code"
|
|
},
|
|
{
|
|
"filename": "ge_doorbell_real.sub",
|
|
"expected_protocol": "GE Doorbell",
|
|
"category": "doorbell",
|
|
"frequency": 433920000,
|
|
"source_path": "Doorbells/GE_Doorbell_19297.sub",
|
|
"notes": "GE wireless doorbell"
|
|
},
|
|
{
|
|
"filename": "byron_doorbell_real.sub",
|
|
"expected_protocol": "Byron Doorbell",
|
|
"category": "doorbell",
|
|
"frequency": 433920000,
|
|
"source_path": "Doorbells/Byron/Byron_DB421E.sub",
|
|
"notes": "Byron DB421E doorbell"
|
|
},
|
|
{
|
|
"filename": "rgb_led_remote_real.sub",
|
|
"expected_protocol": "RGB LED Remote",
|
|
"category": "led_controller",
|
|
"frequency": 433920000,
|
|
"source_path": "LED/RGB_LED_Lamp/red.sub",
|
|
"notes": "RGB LED lamp remote control"
|
|
},
|
|
{
|
|
"filename": "ceiling_fan_real.sub",
|
|
"expected_protocol": "Ceiling Fan Remote",
|
|
"category": "fan_controller",
|
|
"frequency": 433920000,
|
|
"source_path": "Ceiling_Fans/Unknown_Fan_CHQ8BT7053T/Fan_1.sub",
|
|
"notes": "Unknown ceiling fan remote"
|
|
},
|
|
{
|
|
"filename": "ceiling_fan2_real.sub",
|
|
"expected_protocol": "Ceiling Fan Remote",
|
|
"category": "fan_controller",
|
|
"frequency": 433920000,
|
|
"source_path": "Ceiling_Fans/Unknown_Fan_CHQ8BT7053T/Fan_2.sub",
|
|
"notes": "Unknown ceiling fan remote (button 2)"
|
|
},
|
|
{
|
|
"filename": "ceiling_fan_light_real.sub",
|
|
"expected_protocol": "Ceiling Fan Light",
|
|
"category": "fan_controller",
|
|
"frequency": 433920000,
|
|
"source_path": "Ceiling_Fans/Unknown_Fan_CHQ8BT7053T/Light_2.sub",
|
|
"notes": "Ceiling fan light control"
|
|
}
|
|
]
|
|
}
|