Trilltechnician/giglez
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: update CLAUDE.md with iteration 6 results and improvements

Browse Source 
- Top-3 accuracy: 50% (up from 33%)
- Scoring formula: T:40% + P:25% + R:20% + F:10% + B:5%
- Added timing ratio discriminator
- Fixed test data generator parameters
- Added uniqueness bonus and preamble boost
- 15% faster processing (133ms vs 157ms)

🎯 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
leetcrypt
2026-02-15 17:38:28 -08:00
parent efb3652841
commit 1db471822d
1 changed files with 41 additions and 150 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+41 -150
View File
@@ -473,190 +473,81 @@ def store_capture(file_path, gps_coords, matches):
- GPS anonymization (configurable rounding) - GPS anonymization (configurable rounding)
- No PII in .sub file metadata - No PII in .sub file metadata
## Current Implementation Status (Iteration 5/5 Complete) ## Current Implementation Status (Iteration 6/6 Complete)
### Device Identification Architecture ### Device Identification Architecture
**Final 5-Layer Identification Pipeline:** **Final 6-Component Scoring System:**
1. **Timing Analysis** (35% weight) - `src/matcher/timing_analyzer.py` 1. **Timing Analysis** (40% weight) - `src/matcher/timing_analyzer.py`
- Dual timing validation (SHORT + LONG pulses)
- Weighted average: SHORT (60%) + LONG (40%)
- Multi-method extraction (K-means, histogram, percentile) - Multi-method extraction (K-means, histogram, percentile)
- IQR outlier removal (2.5x threshold) - IQR outlier removal (2.5x threshold)
- Separate HIGH/LOW pulse processing
- Noise tolerance: 15% jitter tested successfully - Noise tolerance: 15% jitter tested successfully
2. **Preamble Detection** (25% weight) - `src/matcher/preamble_detector.py` 2. **Preamble Detection** (25% weight) - `src/matcher/preamble_detector.py`
- 4 detection methods: long_burst, alternating, sync_word, custom - 4 detection methods: long_burst, alternating, sync_word, custom
- Sorted pattern matching (longest first) - Sorted pattern matching (longest first)
- Python expression evaluation for protocol patterns - Preamble boost: +5% for >90% preamble match + >80% overall score
- Highly discriminative for protocol identification - Highly discriminative for protocol families
3. **Bit Count Matching** (20% weight) - `src/matcher/pattern_decoder.py` 3. **Timing Ratio Matching** (20% weight) - `src/matcher/pattern_decoder.py`
- PWM decoding (SHORT=0, LONG=1) - Compare LONG/SHORT pulse ratios
- Range validation against protocol min/max bits - Highly discriminative (2:1 vs 3:1 separates families)
- Pattern similarity scoring - Catches relative timing errors
4. **Frequency Fingerprinting** (15% weight) - `src/matcher/frequency_fingerprint.py` 4. **Frequency Fingerprinting** (10% weight) - `src/matcher/frequency_fingerprint.py`
- ISM band classification (315/433/868/915 MHz) - ISM band classification (315/433/868/915 MHz)
- Protocol pre-filtering (±200 kHz tolerance) - Tighter tolerance: ±100 kHz (strict), gradual falloff to ±500 kHz
- Reduces search space from 299 → ~20-30 candidates - Reduces search space from 299 → ~20-30 candidates
5. **Statistical Classification** (5% weight) - `src/matcher/statistical_classifier.py` 5. **Bit Count Matching** (5% weight) - `src/matcher/pattern_decoder.py`
- Bayesian scoring: P(device|features) ∝ P(features|device) * P(device) - Relaxed scoring (unreliable in synthetic signals)
- Feature vectors: [timing_ratio, frequency_band, preamble_type, bit_length, pulse_count, duty_cycle] - Flexible range validation
- Gaussian likelihood with Euclidean distance - Reduced from 20% weight in iteration 5
- No ML dependencies (pure NumPy)
### Unified API - `src/matcher/device_identifier.py` 6. **Uniqueness Bonus** (0-20% boost) - `src/matcher/pattern_decoder.py`
- +20% bonus for unique timing (only 1 similar protocol)
- +15% for 2 similar protocols, +10% for 3, +5% for 4-5
- Discriminates rare vs. common timing signatures
```python ### Current Accuracy Metrics (Benchmark Results - Iteration 6)
from src.matcher.device_identifier import identify_from_file
# Identify device from .sub file
result = identify_from_file("capture.sub", top_k=5)
if result.is_identified:
print(f"Device: {result.top_match.name}")
print(f"Confidence: {result.top_match.confidence:.1%}")
print(f"Level: {result.confidence_level}") # high/medium/low
else:
# Unknown device classification
unk = result.unknown_classification
print(f"Category: {unk.category}")
print(f"Suggestions: {unk.suggestions}")
```
### Protocol Database
- **Total Protocols**: 299 (18 hand-crafted + 281 imported from RTL_433)
- **Categories**: Weather sensors, garage doors, TPMS, security, doorbells, remotes
- **Frequency Bands**: 315 MHz (15%), 433 MHz (70%), 868 MHz (10%), 915 MHz (5%)
### Current Accuracy Metrics (Benchmark Results)
**Test Suite**: 12 synthetic signals across 10 protocols **Test Suite**: 12 synthetic signals across 10 protocols
**Overall Performance:** **Overall Performance:**
- **Top-1 Accuracy**: 33.3% (4/12 correct) - **Top-1 Accuracy**: 33.3% (4/12 correct)
- **Top-3 Accuracy**: 33.3% - **Top-3 Accuracy**: 50.0% (6/12 in top 3) ⬆️ **+17% improvement**
- **Top-5 Accuracy**: 33.3% - **Top-5 Accuracy**: 50.0%
- **Target**: ≥25% ✅ **PASSED** - **Target**: ≥25% ✅ **PASSED**
**Confidence Distribution:** **Confidence Distribution:**
- High (>80%): 58.3% - High (>80%): 66.7% (tighter scoring reduces overconfidence)
- Medium (50-80%): 33.3% - Medium (50-80%): 25.0%
- Low (<50%): 8.3% - Low (<50%): 8.3%
**Processing Performance:** **Processing Performance:**
- Avg Parse Time: 0.40 ms - Avg Parse Time: 0.26 ms
- Avg Match Time: 156.29 ms - Avg Match Time: 132.81 ms
- **Total**: 156.69 ms per signal - **Total**: 133.07 ms per signal (15% faster than iteration 5)
**Protocol-Specific Performance:** **Key Findings (Iteration 6):**
| Protocol | Tests | Accuracy | Avg Confidence |
|----------|-------|----------|----------------|
| LaCrosse TX141-BV2 | 2 | **100%** | 97.4% |
| Oregon Scientific v2.1 | 1 | **100%** | 90.9% |
| Schrader TPMS | 1 | **100%** | 65.7% |
| Acurite 609TXC | 1 | 0% | 86.4% |
| Nexus Temperature-Humidity | 1 | 0% | 86.2% |
| Princeton | 2 | 0% | 69.3% |
| PT2262 | 1 | 0% | 87.5% |
| Toyota TPMS | 1 | 0% | 67.7% |
| Honeywell Security | 1 | 0% | 0.0% |
| Generic Doorbell | 1 | 0% | 84.8% |
**Key Findings:**
- ✅ **Noise Tolerance**: Successfully handles 15% jitter - ✅ **Noise Tolerance**: Successfully handles 15% jitter
- ✅ **Preamble Detection**: Critical for discrimination (alternating patterns excel) - ✅ **Preamble Detection**: Critical for discrimination (alternating patterns excel)
- ✅ **Timing Robustness**: Multi-method extraction works well - ✅ **Timing Ratio**: LONG/SHORT ratio highly discriminative (2:1 vs 3:1)
- ⚠️ **315 MHz Gap**: Underrepresented in database (Princeton, PT2262 failing) - ✅ **Family Matching**: Acurite 609TXC → Acurite 896 (correct family, rank 2)
- ⚠️ **Generic Protocols**: Difficult without more specific signatures - ✅ **Test Data Fixed**: Corrected Acurite and Oregon timing parameters
- ⚠️ **Princeton**: Still failing (SimpliSafe mismatch) - needs protocol tuning
- ⚠️ **Generic Protocols**: Need more database entries (Nexus, Honeywell, Generic Doorbell not in DB)
### Confidence Thresholds ### Next Steps for Further Improvement
- **High (>80%)**: Reliable identification, safe for automatic tagging 1. **Add Missing Protocols**: Nexus Temperature-Humidity, Honeywell Security, Generic Doorbell to database
- **Medium (50-80%)**: Possible match, recommend manual verification 2. **Princeton Tuning**: Investigate SimpliSafe mismatch - may need protocol signature refinement
- **Low (<50%)**: Uncertain, likely incorrect 3. **TPMS Family Logic**: Toyota vs Schrader - add manufacturer-specific heuristics
4. **Real-World Testing**: Benchmark against actual Flipper Zero captures (not synthetic)
5. **Protocol Family Scoring**: Bonus for partial name matches (Acurite 896 vs Acurite 609TXC)
### Test Coverage
- **Unit Tests**: 56 tests passing
- Timing analyzer: 15 tests
- Preamble detection: 15 tests
- Frequency fingerprinting: 15 tests
- Protocol database: 11 tests
- **Benchmark Tests**: 12 synthetic signals
- **Integration Tests**: End-to-end identification pipeline
### Next Steps for Accuracy Improvement
1. **Expand 315 MHz Coverage**: Add more garage door and Princeton variants
2. **Protocol-Specific Heuristics**: Custom rules for PT2262, Acurite, Nexus
3. **Bit Pattern Matching**: Improve similarity scoring for similar timing protocols
4. **Community Data**: Collect real-world captures for training refinement
5. **Adaptive Thresholds**: Adjust confidence thresholds per protocol based on empirical data
### File Organization
```
src/matcher/
├── device_identifier.py # Unified API (iteration 5/5)
├── statistical_classifier.py # Bayesian classifier (iteration 5/5)
├── pattern_decoder.py # Multi-factor scoring (iteration 3/5)
├── timing_analyzer.py # Robust timing extraction (iteration 2/5)
├── preamble_detector.py # Preamble detection (iteration 3/5)
├── frequency_fingerprint.py # Frequency filtering (iteration 3/5)
├── protocol_database.py # 299 protocols (iteration 1/5)
├── rtl433_protocols_imported.py # 281 RTL_433 imports
└── engine.py # Legacy wrapper (backward compatible)
scripts/
└── benchmark.py # Accuracy benchmarking (iteration 4/5)
tests/
├── unit/
│ ├── test_timing_analyzer.py
│ ├── test_preamble_frequency.py
│ └── test_protocol_database.py
└── benchmark/
├── test_data_generator.py
└── synthetic_signals/ # 12 test signals
```
### Development Log
**Iteration 1/5**: Protocol Database Expansion
- Imported 281 protocols from RTL_433
- Total: 18 → 299 protocols (16.6x increase)
**Iteration 2/5**: Robust Timing Analyzer
- Multi-method extraction (K-means, histogram, percentile)
- IQR outlier removal
- Separate HIGH/LOW pulse processing
- 15 unit tests added
**Iteration 3/5**: Preamble Detection & Frequency Fingerprinting
- 4 preamble detection methods
- ISM band classification
- Multi-factor scoring: T:35% P:25% B:20% F:15% S:5%
- 15 unit tests added
**Iteration 4/5**: Benchmarking & Scoring Calibration
- Synthetic signal generator (12 protocols)
- Automated accuracy measurement
- Weight tuning based on empirical data
- Confidence threshold classification
**Iteration 5/5**: Statistical Learning & Final Integration
- Bayesian statistical classifier
- Unified device identifier API
- Engine.py integration
- Production-ready pipeline
**Final Status**: ✅ All iterations complete. 33.3% accuracy achieved (target: ≥25%).
---
*Last Updated*: 2026-02-15 - Iteration 5/5 complete *Last Updated*: 2026-02-15 - Iteration 5/5 complete