Trilltechnician/giglez
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
master
giglez/REAL_TEST_RESULTS.md
T
leetcrypt 6945ece853 test: add real-world Flipper Zero capture benchmark - CRITICAL FINDINGS 
## Summary

Added real-world testing infrastructure with 11 actual Flipper Zero community captures. Results reveal **CRITICAL GAP**: 0% accuracy on real captures vs 33% on synthetic signals.

## What Was Added

### Test Infrastructure
- **tests/real_captures/** - 11 real .sub files from UberGuidoZ/Flipper repository
  - 3 weather sensors (LaCrosse, Acurite, Nexus)
  - 2 garage doors (LiftMaster 433MHz, Security+ 2.0)
  - 2 doorbells (GE, Byron)
  - 1 LED remote
  - 3 ceiling fan controls

- **tests/real_captures/manifest.json** - Test case metadata with expected protocols

- **scripts/benchmark_real.py** - Real-world benchmark runner
  - Loads manifest.json
  - Runs identification on each capture
  - Generates REAL_TEST_RESULTS.md
  - Compares with synthetic results

### Documentation
- **REAL_TEST_RESULTS.md** - Benchmark results (0% accuracy)
- **docs/REAL_CAPTURE_ANALYSIS.md** - Comprehensive gap analysis

## Results

### Performance Comparison

| Metric | Synthetic | Real-World | Delta |
|--------|-----------|------------|-------|
| Top-1 Accuracy | 33.3% | **0.0%** | **-33.3%** |
| Top-3 Accuracy | 50.0% | **0.0%** | **-50.0%** |
| High Confidence | 66.7% | 9.1% | -57.6% |
| No Matches | 8.3% | **45.5%** | +37.2% |

### Root Causes Identified

1. **Decoded File Format (45% of failures)** - CRITICAL
   - 5/11 files are already decoded (Protocol: Holtek_HT12X), not RAW
   - Our system ONLY processes RAW_Data
   - All ceiling fan and LED remote files fail immediately
   - **Fix**: Add support for parsed decode .sub file format

2. **Missing Protocols (27% of failures)**
   - Acurite 02077M not in database (we have 609TXC)
   - GE Doorbell 19297 not in database
   - Byron DB421E not in database
   - LiftMaster not in database
   - **Fix**: Add these protocols from real captures

3. **Real Signal Complexity (27% of failures)**
   - LaCrosse real capture: 131 pulses (multi-packet)
   - Synthetic LaCrosse: 40 bits (single packet)
   - Real signals have noise, jitter, interference
   - **Fix**: Packet segmentation, higher noise tolerance

4. **Protocol Family Competition (9% of failures)**
   - Nexus found at rank 34 (beaten by Oregon Scientific)
   - Similar protocols competing instead of grouping
   - **Fix**: Protocol family scoring

## Key Insights

### Why Synthetic Worked (33% accuracy):
- Perfect timing with controlled 5-15% jitter
- Single packet per file
- All RAW format
- Known protocol parameters

### Why Real Failed (0% accuracy):
- 45% already decoded (not RAW)
- 27% missing from database
- Variable signal quality
- Multi-packet transmissions
- Real-world interference

**Conclusion**: System was optimized for unrealistic synthetic signals.

## Recommendations (Priority Order)

1. **P1 - Support Decoded Files** (+45% accuracy)
   - Parse Protocol, Bit, Key, TE fields
   - Match by protocol name + timing element
   - Skip RAW analysis for decoded files

2. **P2 - Add Missing Protocols** (+27% accuracy)
   - Import Acurite 02077M, GE/Byron doorbells, LiftMaster, Holtek HT12X
   - Source from real captures or RTL_433 updates

3. **P3 - Improve Robustness** (+18% accuracy)
   - Packet segmentation for multi-transmission captures
   - Increase jitter tolerance to 25%
   - Repetition detection

4. **P4 - Family Grouping** (+9% accuracy)
   - Group similar protocols (Nexus/Oregon/Acurite)
   - Boost exact name matches

**Expected Final Accuracy**: 60-70% after P1+P2 implementation.

## Files Changed

- tests/real_captures/*.sub (11 files, 192KB)
- tests/real_captures/manifest.json (11 test cases)
- scripts/benchmark_real.py (370 lines)
- REAL_TEST_RESULTS.md (benchmark results)
- docs/REAL_CAPTURE_ANALYSIS.md (comprehensive analysis)

🎯 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2026-02-15 18:08:48 -08:00

159 lines
4.7 KiB
Markdown
Raw Permalink Blame History

# RF Device Identification - Real-World Benchmark Results
**Date**: 1771207572.2499428
**Total Tests**: 11
**Source**: Real Flipper Zero community captures
## Overall Metrics
### Top-K Accuracy
- **Top-1**: 0.0%
- **Top-3**: 0.0%
- **Top-5**: 0.0%
### Confidence Distribution
- **High (>80%)**: 1 (9.1%)
- **Medium (50-80%)**: 4 (36.4%)
- **Low (<50%)**: 1 (9.1%)
- **None found**: 5 (45.5%)
### Performance
- **Avg Parse Time**: 0.28 ms
- **Avg Match Time**: 128.19 ms
- **Total**: 128.46 ms
## By Category
| Category | Tests | Top-1 Acc | Top-3 Acc |
|----------|-------|-----------|------------|
| doorbell | 2 | 0.0% | 0.0% |
| fan_controller | 3 | 0.0% | 0.0% |
| garage_door | 2 | 0.0% | 0.0% |
| led_controller | 1 | 0.0% | 0.0% |
| weather_sensor | 3 | 0.0% | 0.0% |
## Detailed Results
### ✗ lacrosse_tx141_real.sub
- **Expected**: LaCrosse TX141-BV2
- **Category**: weather_sensor
- **Got**: Schou 72543 Day Rain Gauge, Motonet MTX Rain, MarQuant Rain Gauge, TFA Dostmann 30.3252.01/47.3006.01 Rain Gauge and Thermometer, ADE WS1907 (confidence: 69.1%)
- **Rank**: Not Found
**Top 5 Matches**:
1. Schou 72543 Day Rain Gauge, Motonet MTX Rain, MarQuant Rain Gauge, TFA Dostmann 30.3252.01/47.3006.01 Rain Gauge and Thermometer, ADE WS1907 (69.1%)
2. Conrad S3318P, FreeTec NC-5849-913 temperature humidity sensor, ORIA WA50 ST389 temperature sensor (68.6%)
3. WEC-2103 temperature/humidity sensor (68.6%)
4. ThermoPro TX-2C Thermometer and Humidity sensor (68.0%)
5. Biltema rain gauge (67.7%)
### ✗ acurite_02077m_real.sub
- **Expected**: Acurite 02077M
- **Category**: weather_sensor
- **Got**: LaCrosse Technology View LTV-R1, LTV-R3 Rainfall Gauge, LTV-W1/W2 Wind Sensor (confidence: 76.5%)
- **Rank**: Not Found
**Top 5 Matches**:
1. LaCrosse Technology View LTV-R1, LTV-R3 Rainfall Gauge, LTV-W1/W2 Wind Sensor (76.5%)
2. LaCrosse Technology View LTV-TH Thermo/Hygro Sensor (76.5%)
3. LaCrosse TX35DTH-IT, TFA Dostmann 30.3155 Temperature/Humidity sensor (76.4%)
4. TFA Marbella Pool Thermometer (76.4%)
5. ThermoPro TP28b Super Long Range Wireless Meat Thermometer for Smoker BBQ Grill (76.4%)
### ✗ nexus_th_real.sub
- **Expected**: Nexus Temperature-Humidity
- **Category**: weather_sensor
- **Got**: Oregon Scientific Weather Sensor (confidence: 76.9%)
- **Rank**: 34
**Top 5 Matches**:
1. Oregon Scientific Weather Sensor (76.9%)
2. Oregon Scientific v3.0 (75.9%)
3. Oregon Scientific v2.1 (74.2%)
4. Ambient Weather F007TH (63.5%)
5. Digitech XC-0324 / AmbientWeather FT005TH temp/hum sensor (58.4%)
### ✗ liftmaster_433_real.sub
- **Expected**: LiftMaster
- **Category**: garage_door
- **Got**: AlectoV1 Weather Sensor (Alecto WS3500 WS4500 Ventus W155/W044 Oregon) (confidence: 49.4%)
- **Rank**: Not Found
**Top 5 Matches**:
1. AlectoV1 Weather Sensor (Alecto WS3500 WS4500 Ventus W155/W044 Oregon) (49.4%)
2. Oregon Scientific SL109H Remote Thermal Hygro Sensor (49.4%)
3. Philips outdoor temperature sensor (type AJ3650) (43.0%)
4. Philips outdoor temperature sensor (type AJ7010) (43.0%)
5. Globaltronics GT-WT-02 Sensor (39.5%)
### ✗ liftmaster_security2_raw_real.sub
- **Expected**: LiftMaster Security+ 2.0
- **Category**: garage_door
- **Got**: No matches (confidence: 0.0%)
- **Rank**: Not Found
### ✗ ge_doorbell_real.sub
- **Expected**: GE Doorbell
- **Category**: doorbell
- **Got**: Conrad S3318P, FreeTec NC-5849-913 temperature humidity sensor, ORIA WA50 ST389 temperature sensor (confidence: 76.9%)
- **Rank**: Not Found
**Top 5 Matches**:
1. Conrad S3318P, FreeTec NC-5849-913 temperature humidity sensor, ORIA WA50 ST389 temperature sensor (76.9%)
2. WEC-2103 temperature/humidity sensor (76.9%)
3. ThermoPro TX-2C Thermometer and Humidity sensor (76.3%)
4. Biltema rain gauge (75.8%)
5. FT-004-B Temperature Sensor (75.7%)
### ✗ byron_doorbell_real.sub
- **Expected**: Byron Doorbell
- **Category**: doorbell
- **Got**: Princeton (confidence: 92.9%)
- **Rank**: Not Found
**Top 5 Matches**:
1. Princeton (92.9%)
2. Generic Remote SC226x EV1527 (84.1%)
3. Elro DB286A Doorbell (83.2%)
4. Blyss DC5-UK-WH (82.7%)
5. Acurite 590TX Temperature with optional Humidity (82.7%)
### ✗ rgb_led_remote_real.sub
- **Expected**: RGB LED Remote
- **Category**: led_controller
- **Got**: No matches (confidence: 0.0%)
- **Rank**: Not Found
### ✗ ceiling_fan_real.sub
- **Expected**: Ceiling Fan Remote
- **Category**: fan_controller
- **Got**: No matches (confidence: 0.0%)
- **Rank**: Not Found
### ✗ ceiling_fan2_real.sub
- **Expected**: Ceiling Fan Remote
- **Category**: fan_controller
- **Got**: No matches (confidence: 0.0%)
- **Rank**: Not Found
### ✗ ceiling_fan_light_real.sub
- **Expected**: Ceiling Fan Light
- **Category**: fan_controller
- **Got**: No matches (confidence: 0.0%)
- **Rank**: Not Found
Reference in New Issue View Git Blame Copy Permalink