 leetcrypt
|
6945ece853
|
test: add real-world Flipper Zero capture benchmark - CRITICAL FINDINGS
## Summary
Added real-world testing infrastructure with 11 actual Flipper Zero community captures. Results reveal **CRITICAL GAP**: 0% accuracy on real captures vs 33% on synthetic signals.
## What Was Added
### Test Infrastructure
- **tests/real_captures/** - 11 real .sub files from UberGuidoZ/Flipper repository
- 3 weather sensors (LaCrosse, Acurite, Nexus)
- 2 garage doors (LiftMaster 433MHz, Security+ 2.0)
- 2 doorbells (GE, Byron)
- 1 LED remote
- 3 ceiling fan controls
- **tests/real_captures/manifest.json** - Test case metadata with expected protocols
- **scripts/benchmark_real.py** - Real-world benchmark runner
- Loads manifest.json
- Runs identification on each capture
- Generates REAL_TEST_RESULTS.md
- Compares with synthetic results
### Documentation
- **REAL_TEST_RESULTS.md** - Benchmark results (0% accuracy)
- **docs/REAL_CAPTURE_ANALYSIS.md** - Comprehensive gap analysis
## Results
### Performance Comparison
| Metric | Synthetic | Real-World | Delta |
|--------|-----------|------------|-------|
| Top-1 Accuracy | 33.3% | **0.0%** | **-33.3%** |
| Top-3 Accuracy | 50.0% | **0.0%** | **-50.0%** |
| High Confidence | 66.7% | 9.1% | -57.6% |
| No Matches | 8.3% | **45.5%** | +37.2% |
### Root Causes Identified
1. **Decoded File Format (45% of failures)** - CRITICAL
- 5/11 files are already decoded (Protocol: Holtek_HT12X), not RAW
- Our system ONLY processes RAW_Data
- All ceiling fan and LED remote files fail immediately
- **Fix**: Add support for parsed decode .sub file format
2. **Missing Protocols (27% of failures)**
- Acurite 02077M not in database (we have 609TXC)
- GE Doorbell 19297 not in database
- Byron DB421E not in database
- LiftMaster not in database
- **Fix**: Add these protocols from real captures
3. **Real Signal Complexity (27% of failures)**
- LaCrosse real capture: 131 pulses (multi-packet)
- Synthetic LaCrosse: 40 bits (single packet)
- Real signals have noise, jitter, interference
- **Fix**: Packet segmentation, higher noise tolerance
4. **Protocol Family Competition (9% of failures)**
- Nexus found at rank 34 (beaten by Oregon Scientific)
- Similar protocols competing instead of grouping
- **Fix**: Protocol family scoring
## Key Insights
### Why Synthetic Worked (33% accuracy):
- Perfect timing with controlled 5-15% jitter
- Single packet per file
- All RAW format
- Known protocol parameters
### Why Real Failed (0% accuracy):
- 45% already decoded (not RAW)
- 27% missing from database
- Variable signal quality
- Multi-packet transmissions
- Real-world interference
**Conclusion**: System was optimized for unrealistic synthetic signals.
## Recommendations (Priority Order)
1. **P1 - Support Decoded Files** (+45% accuracy)
- Parse Protocol, Bit, Key, TE fields
- Match by protocol name + timing element
- Skip RAW analysis for decoded files
2. **P2 - Add Missing Protocols** (+27% accuracy)
- Import Acurite 02077M, GE/Byron doorbells, LiftMaster, Holtek HT12X
- Source from real captures or RTL_433 updates
3. **P3 - Improve Robustness** (+18% accuracy)
- Packet segmentation for multi-transmission captures
- Increase jitter tolerance to 25%
- Repetition detection
4. **P4 - Family Grouping** (+9% accuracy)
- Group similar protocols (Nexus/Oregon/Acurite)
- Boost exact name matches
**Expected Final Accuracy**: 60-70% after P1+P2 implementation.
## Files Changed
- tests/real_captures/*.sub (11 files, 192KB)
- tests/real_captures/manifest.json (11 test cases)
- scripts/benchmark_real.py (370 lines)
- REAL_TEST_RESULTS.md (benchmark results)
- docs/REAL_CAPTURE_ANALYSIS.md (comprehensive analysis)
🎯 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
|
2026-02-15 18:08:48 -08:00 |
|