Trilltechnician/giglez
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

test: add real-world Flipper Zero capture benchmark - CRITICAL FINDINGS

Browse Source 
## Summary

Added real-world testing infrastructure with 11 actual Flipper Zero community captures. Results reveal **CRITICAL GAP**: 0% accuracy on real captures vs 33% on synthetic signals.

## What Was Added

### Test Infrastructure
- **tests/real_captures/** - 11 real .sub files from UberGuidoZ/Flipper repository
  - 3 weather sensors (LaCrosse, Acurite, Nexus)
  - 2 garage doors (LiftMaster 433MHz, Security+ 2.0)
  - 2 doorbells (GE, Byron)
  - 1 LED remote
  - 3 ceiling fan controls

- **tests/real_captures/manifest.json** - Test case metadata with expected protocols

- **scripts/benchmark_real.py** - Real-world benchmark runner
  - Loads manifest.json
  - Runs identification on each capture
  - Generates REAL_TEST_RESULTS.md
  - Compares with synthetic results

### Documentation
- **REAL_TEST_RESULTS.md** - Benchmark results (0% accuracy)
- **docs/REAL_CAPTURE_ANALYSIS.md** - Comprehensive gap analysis

## Results

### Performance Comparison

| Metric | Synthetic | Real-World | Delta |
|--------|-----------|------------|-------|
| Top-1 Accuracy | 33.3% | **0.0%** | **-33.3%** |
| Top-3 Accuracy | 50.0% | **0.0%** | **-50.0%** |
| High Confidence | 66.7% | 9.1% | -57.6% |
| No Matches | 8.3% | **45.5%** | +37.2% |

### Root Causes Identified

1. **Decoded File Format (45% of failures)** - CRITICAL
   - 5/11 files are already decoded (Protocol: Holtek_HT12X), not RAW
   - Our system ONLY processes RAW_Data
   - All ceiling fan and LED remote files fail immediately
   - **Fix**: Add support for parsed decode .sub file format

2. **Missing Protocols (27% of failures)**
   - Acurite 02077M not in database (we have 609TXC)
   - GE Doorbell 19297 not in database
   - Byron DB421E not in database
   - LiftMaster not in database
   - **Fix**: Add these protocols from real captures

3. **Real Signal Complexity (27% of failures)**
   - LaCrosse real capture: 131 pulses (multi-packet)
   - Synthetic LaCrosse: 40 bits (single packet)
   - Real signals have noise, jitter, interference
   - **Fix**: Packet segmentation, higher noise tolerance

4. **Protocol Family Competition (9% of failures)**
   - Nexus found at rank 34 (beaten by Oregon Scientific)
   - Similar protocols competing instead of grouping
   - **Fix**: Protocol family scoring

## Key Insights

### Why Synthetic Worked (33% accuracy):
- Perfect timing with controlled 5-15% jitter
- Single packet per file
- All RAW format
- Known protocol parameters

### Why Real Failed (0% accuracy):
- 45% already decoded (not RAW)
- 27% missing from database
- Variable signal quality
- Multi-packet transmissions
- Real-world interference

**Conclusion**: System was optimized for unrealistic synthetic signals.

## Recommendations (Priority Order)

1. **P1 - Support Decoded Files** (+45% accuracy)
   - Parse Protocol, Bit, Key, TE fields
   - Match by protocol name + timing element
   - Skip RAW analysis for decoded files

2. **P2 - Add Missing Protocols** (+27% accuracy)
   - Import Acurite 02077M, GE/Byron doorbells, LiftMaster, Holtek HT12X
   - Source from real captures or RTL_433 updates

3. **P3 - Improve Robustness** (+18% accuracy)
   - Packet segmentation for multi-transmission captures
   - Increase jitter tolerance to 25%
   - Repetition detection

4. **P4 - Family Grouping** (+9% accuracy)
   - Group similar protocols (Nexus/Oregon/Acurite)
   - Boost exact name matches

**Expected Final Accuracy**: 60-70% after P1+P2 implementation.

## Files Changed

- tests/real_captures/*.sub (11 files, 192KB)
- tests/real_captures/manifest.json (11 test cases)
- scripts/benchmark_real.py (370 lines)
- REAL_TEST_RESULTS.md (benchmark results)
- docs/REAL_CAPTURE_ANALYSIS.md (comprehensive analysis)

🎯 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
leetcrypt
2026-02-15 18:08:39 -08:00
parent 1db471822d
commit 6945ece853
15 changed files with 1126 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/REAL_CAPTURE_ANALYSIS.md
+298
View File
@@ -0,0 +1,298 @@
# Real-World Capture Analysis - Gap Between Synthetic and Real Performance
**Date**: 2026-02-15
**Benchmark**: Real Flipper Zero Community Captures vs Synthetic Test Signals
## Executive Summary
**CRITICAL FINDING**: The RF device identification system achieves **0% accuracy on real-world captures** vs **33% on synthetic signals**. This represents a complete failure to generalize from synthetic to real-world data.
## Performance Comparison
| Metric | Synthetic | Real-World | Delta |
|--------|-----------|------------|-------|
| Top-1 Accuracy | 33.3% (4/12) | **0.0%** (0/11) | **-33.3%** |
| Top-3 Accuracy | 50.0% (6/12) | **0.0%** (0/11) | **-50.0%** |
| High Confidence (>80%) | 66.7% | **9.1%** | -57.6% |
| No Matches Found | 8.3% (1/12) | **45.5%** (5/11) | +37.2% |
## Root Causes
### 1. **Decoded vs RAW File Format** (5/11 failures = 45%)
**Problem**: Many real Flipper captures are **already decoded**, not RAW.
**Evidence**:
- All 3 ceiling fan files: `Protocol: Holtek_HT12X` (decoded)
- RGB LED remote: Already decoded
- LiftMaster Security+ 2.0: Already decoded
**Example** (`ceiling_fan_real.sub`):
```
Filetype: Flipper SubGhz Key File
Version: 1
Frequency: 315000000
Preset: FuriHalSubGhzPresetOok650Async
Protocol: Holtek_HT12X # <-- Already decoded!
Bit: 12
Key: 00 00 00 00 00 00 00 BF
TE: 470
```
**Impact**: Our system **only processes RAW_Data**. Decoded files have `has_raw_data=False`, so the pattern decoder immediately returns empty results.
**Fix Required**: Add support for decoded .sub files:
1. Parse `Protocol`, `Bit`, `Key`, `TE` fields
2. Match against database by protocol name + timing element
3. Synthesize timing from TE (timing element) if needed
---
### 2. **Real Signal Noise & Complexity** (3/11 failures = 27%)
**Problem**: Real captures have:
- Variable signal quality
- Multi-packet transmissions
- Long pulse trains (LaCrosse: 131 pulses total)
- Interference and jitter beyond our 15% tolerance
**Evidence**:
- `lacrosse_tx141_real.sub`: 131 pulses, very long RAW_Data
- Top match: "Schou 72543 Day Rain Gauge" (69.1%) - WRONG
- Expected "LaCrosse TX141-BV2" not found in top 100
**Analysis**: Real LaCrosse signals are much longer and noisier than our 40-bit synthetic version.
---
### 3. **Missing Protocols in Database** (3/11 failures = 27%)
**Problem**: Expected protocols not in our 299-protocol database:
- "Acurite 02077M" → Not in database (we have Acurite 609TXC, but not 02077M variant)
- "GE Doorbell 19297" → Not in database
- "Byron Doorbell DB421E" → Not in database (Byron got Princeton at 92.9%)
**Evidence**:
```
Byron doorbell → Princeton (92.9% confidence)
GE doorbell → Conrad S3318P sensor (76.9% confidence)
```
**Impact**: Even with perfect timing extraction, we can't match protocols that don't exist in the database.
---
### 4. **Protocol Name Matching Too Strict** (1/11 = 9%)
**Problem**: `nexus_th_real.sub` matched "Nexus Temperature-Humidity" but at **rank 34**.
**Top 5 Results**:
1. Oregon Scientific Weather Sensor (76.9%)
2. Oregon Scientific v3.0 (75.9%)
3. Oregon Scientific v2.1 (74.2%)
4. Ambient Weather F007TH (63.5%)
5. Digitech XC-0324 (58.4%)
...
34. Nexus Temperature-Humidity
**Analysis**: Nexus is getting matched, but Oregon protocols score higher. This suggests:
- Timing similarity between Nexus and Oregon
- Preamble detection favoring Oregon
- Need protocol-family grouping (Nexus vs Oregon shouldn't compete directly)
---
## Detailed Failure Analysis
### Weather Sensors (0/3 correct)
#### 1. LaCrosse TX141-BV2 Real Capture
- **Expected**: LaCrosse TX141-BV2
- **Got**: Schou 72543 Day Rain Gauge (69.1%)
- **Rank**: Not found
**Issue**: Real capture has 131 pulses vs synthetic 40 bits. Likely multiple transmissions in one capture.
#### 2. Acurite 02077M
- **Expected**: Acurite 02077M
- **Got**: LaCrosse Technology View LTV-R1 (76.5%)
- **Rank**: Not found
**Issue**: Acurite 02077M not in database. We have Acurite 609TXC but not this model.
#### 3. Nexus Temperature-Humidity
- **Expected**: Nexus Temperature-Humidity
- **Got**: Oregon Scientific Weather Sensor (76.9%)
- **Rank**: 34
**Issue**: Protocol exists but ranks low. Oregon and Nexus are similar protocols competing.
---
### Garage Doors (0/2 correct)
#### 1. LiftMaster 433MHz
- **Expected**: LiftMaster
- **Got**: AlectoV1 Weather Sensor (49.4%)
- **Rank**: Not found
**Issue**: LiftMaster not in database. Low confidence (49%) suggests poor timing match.
#### 2. LiftMaster Security+ 2.0
- **Expected**: LiftMaster Security+ 2.0
- **Got**: No matches (0%)
- **Rank**: Not found
**Issue**: **Decoded file format**. No RAW_Data to process. Security+ 2.0 uses rolling codes.
---
### Doorbells (0/2 correct)
#### 1. GE Doorbell 19297
- **Expected**: GE Doorbell
- **Got**: Conrad S3318P sensor (76.9%)
- **Rank**: Not found
**Issue**: GE Doorbell not in database. Misidentified as temperature sensor.
#### 2. Byron Doorbell DB421E
- **Expected**: Byron Doorbell
- **Got**: Princeton (92.9%)
- **Rank**: Not found
**Issue**: Byron not in database. High confidence match to Princeton suggests similar timing (Princeton-like protocol).
---
### LED/Fan Controllers (0/4 correct)
#### All 4 files: No matches (0%)
**Issue**: **All are decoded format**, not RAW. Our system cannot process them.
**Files**:
- `rgb_led_remote_real.sub` - Decoded
- `ceiling_fan_real.sub` - Decoded (Holtek_HT12X)
- `ceiling_fan2_real.sub` - Decoded (Holtek_HT12X)
- `ceiling_fan_light_real.sub` - Decoded (Holtek_HT12X)
---
## Recommendations
### Priority 1: Support Decoded .sub Files (Fixes 45% of failures)
**Implementation**:
```python
def parse_sub_file(filepath):
"""Parse both RAW and decoded .sub files"""
metadata = SignalMetadata()
with open(filepath) as f:
# ... existing parsing ...
# If no RAW_Data but has Protocol field
if not metadata.has_raw_data and metadata.protocol:
# This is a decoded file
# Extract: Protocol, Bit, Key, TE
# Match by protocol name + TE timing
metadata.is_decoded = True
metadata.timing_element = te_value
return metadata
```
**Database Matching**:
- For decoded files, match primarily by `Protocol` field name
- Use `TE` (timing element) as secondary confirmation
- Skip RAW pattern analysis
---
### Priority 2: Add Missing Protocols (Fixes 27% of failures)
**Protocols to Add**:
1. **Acurite 02077M** - Weather station (433.92 MHz)
2. **GE Doorbell 19297** - Doorbell (433.92 MHz)
3. **Byron DB421E** - Doorbell (433.92 MHz)
4. **LiftMaster** (generic 433 MHz) - Garage door
5. **Holtek HT12X** - LED/Fan remote encoder chip
**Source**: Import from real captures or RTL_433 database updates.
---
### Priority 3: Improve Real Signal Robustness (Fixes 27% of failures)
**Issues**:
- Long multi-packet captures (131 pulses vs 40 bits)
- Variable signal quality
- Packet repetition detection needed
**Fixes**:
1. **Packet Segmentation**: Split long RAW_Data into individual transmissions
2. **Repetition Detection**: Find repeating patterns, score best match
3. **Noise Filtering**: Increase jitter tolerance from 15% to 25% for real data
4. **Confidence Calibration**: Lower thresholds for real-world data
---
### Priority 4: Protocol Family Grouping (Fixes 9% of failures)
**Issue**: Nexus ranks #34, beaten by Oregon Scientific variants.
**Fix**:
- Group similar protocols (Nexus, Oregon Scientific, Acurite all use similar encoding)
- Score by protocol family first, then by specific model
- Boost exact name matches over family matches
---
## Performance Expectations After Fixes
| Fix | Impact | Expected Accuracy |
|-----|--------|-------------------|
| Support decoded files | +45% (5/11) | 45% |
| Add missing protocols | +27% (3/11) | 72% |
| Improve robustness | +18% (2/11) | 90% |
| Family grouping | +9% (1/11) | 100% |
**Realistic Target**: 60-70% top-1 accuracy after implementing Priority 1 and 2.
---
## Synthetic vs Real: Why the Gap?
### Synthetic Signals (33% accuracy)
- ✅ Perfect timing (no jitter beyond 5-15%)
- ✅ Single packet per file
- ✅ Known protocol parameters
- ✅ All RAW format
- ❌ Unrealistic simplicity
### Real Captures (0% accuracy)
- ❌ 45% are decoded (not RAW)
- ❌ 27% use protocols not in database
- ❌ Variable signal quality
- ❌ Multi-packet transmissions
- ❌ Real-world interference
- ✅ Actual device behavior
**Conclusion**: Our system was optimized for synthetic signals that don't represent real-world usage.
---
## Next Steps
1. **Immediate**: Implement decoded .sub file support (Priority 1)
2. **Short-term**: Add missing protocols from real captures (Priority 2)
3. **Medium-term**: Improve noise tolerance and packet segmentation (Priority 3)
4. **Long-term**: Collect more real captures for training/validation
---
**Generated**: 2026-02-15
**Benchmark Version**: Iteration 6 (post-precision tuning)
**Database Size**: 299 protocols (RTL_433 imported)
**Real Captures Source**: UberGuidoZ/Flipper repository