feat: benchmark suite + scoring calibration - iteration 4/5
Implements comprehensive benchmarking infrastructure and tunes scoring weights
based on empirical accuracy measurements.
New Components:
- tests/benchmark/test_data_generator.py: Synthetic signal generator for 12 protocols
- scripts/benchmark.py: Full benchmarking suite with accuracy metrics
- TEST_RESULTS_SUMMARY.md: Detailed benchmark results and per-protocol analysis
Benchmark Results:
- Total Tests: 12 synthetic signals across 10 protocols
- Top-1 Accuracy: 33.3% (4/12 correct)
- Top-3 Accuracy: 33.3%
- Confidence Distribution: 66.7% high (>80%), 25% medium (50-80%), 8.3% low (<50%)
- Avg Processing Time: 144.6ms per signal
Scoring Weight Tuning:
BEFORE: Timing(30%) + Frequency(25%) + BitCount(20%) + Preamble(15%) + Stats(10%)
AFTER: Timing(35%) + Preamble(25%) + BitCount(20%) + Frequency(15%) + Stats(5%)
Rationale:
- Increased Preamble weight (15% → 25%): Highly discriminative for protocol identification
- Increased Timing weight (30% → 35%): Core identification feature
- Decreased Frequency weight (25% → 15%): Many protocols share same ISM band
- Decreased Stats weight (10% → 5%): Less discriminative in practice
Confidence Thresholds:
- High: >80% (reliable identification)
- Medium: 50-80% (possible match, needs verification)
- Low: <50% (uncertain, likely incorrect)
Protocol Performance:
✓ Excellent (100%): LaCrosse TX141-BV2, Oregon Scientific v2.1, Schrader TPMS
✗ Needs Improvement (0%): Acurite 609TXC, Princeton, PT2262, Nexus, Toyota TPMS
Key Findings:
- Preamble detection critical for discrimination (alternating patterns work well)
- Timing analysis robust to 15% noise
- 315 MHz protocols underrepresented in database
- Generic protocols difficult to distinguish without more specific signatures
Next Steps (Future Iterations):
- Expand 315 MHz protocol coverage
- Add protocol-specific heuristics for Princeton, PT2262
- Improve bit pattern matching for similar timing protocols
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
leetcrypt
f7329df581