Trilltechnician/giglez
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
master
giglez/REAL_TEST_RESULTS.md
T
leetcrypt 6945ece853 test: add real-world Flipper Zero capture benchmark - CRITICAL FINDINGS 
## Summary

Added real-world testing infrastructure with 11 actual Flipper Zero community captures. Results reveal **CRITICAL GAP**: 0% accuracy on real captures vs 33% on synthetic signals.

## What Was Added

### Test Infrastructure
- **tests/real_captures/** - 11 real .sub files from UberGuidoZ/Flipper repository
  - 3 weather sensors (LaCrosse, Acurite, Nexus)
  - 2 garage doors (LiftMaster 433MHz, Security+ 2.0)
  - 2 doorbells (GE, Byron)
  - 1 LED remote
  - 3 ceiling fan controls

- **tests/real_captures/manifest.json** - Test case metadata with expected protocols

- **scripts/benchmark_real.py** - Real-world benchmark runner
  - Loads manifest.json
  - Runs identification on each capture
  - Generates REAL_TEST_RESULTS.md
  - Compares with synthetic results

### Documentation
- **REAL_TEST_RESULTS.md** - Benchmark results (0% accuracy)
- **docs/REAL_CAPTURE_ANALYSIS.md** - Comprehensive gap analysis

## Results

### Performance Comparison

| Metric | Synthetic | Real-World | Delta |
|--------|-----------|------------|-------|
| Top-1 Accuracy | 33.3% | **0.0%** | **-33.3%** |
| Top-3 Accuracy | 50.0% | **0.0%** | **-50.0%** |
| High Confidence | 66.7% | 9.1% | -57.6% |
| No Matches | 8.3% | **45.5%** | +37.2% |

### Root Causes Identified

1. **Decoded File Format (45% of failures)** - CRITICAL
   - 5/11 files are already decoded (Protocol: Holtek_HT12X), not RAW
   - Our system ONLY processes RAW_Data
   - All ceiling fan and LED remote files fail immediately
   - **Fix**: Add support for parsed decode .sub file format

2. **Missing Protocols (27% of failures)**
   - Acurite 02077M not in database (we have 609TXC)
   - GE Doorbell 19297 not in database
   - Byron DB421E not in database
   - LiftMaster not in database
   - **Fix**: Add these protocols from real captures

3. **Real Signal Complexity (27% of failures)**
   - LaCrosse real capture: 131 pulses (multi-packet)
   - Synthetic LaCrosse: 40 bits (single packet)
   - Real signals have noise, jitter, interference
   - **Fix**: Packet segmentation, higher noise tolerance

4. **Protocol Family Competition (9% of failures)**
   - Nexus found at rank 34 (beaten by Oregon Scientific)
   - Similar protocols competing instead of grouping
   - **Fix**: Protocol family scoring

## Key Insights

### Why Synthetic Worked (33% accuracy):
- Perfect timing with controlled 5-15% jitter
- Single packet per file
- All RAW format
- Known protocol parameters

### Why Real Failed (0% accuracy):
- 45% already decoded (not RAW)
- 27% missing from database
- Variable signal quality
- Multi-packet transmissions
- Real-world interference

**Conclusion**: System was optimized for unrealistic synthetic signals.

## Recommendations (Priority Order)

1. **P1 - Support Decoded Files** (+45% accuracy)
   - Parse Protocol, Bit, Key, TE fields
   - Match by protocol name + timing element
   - Skip RAW analysis for decoded files

2. **P2 - Add Missing Protocols** (+27% accuracy)
   - Import Acurite 02077M, GE/Byron doorbells, LiftMaster, Holtek HT12X
   - Source from real captures or RTL_433 updates

3. **P3 - Improve Robustness** (+18% accuracy)
   - Packet segmentation for multi-transmission captures
   - Increase jitter tolerance to 25%
   - Repetition detection

4. **P4 - Family Grouping** (+9% accuracy)
   - Group similar protocols (Nexus/Oregon/Acurite)
   - Boost exact name matches

**Expected Final Accuracy**: 60-70% after P1+P2 implementation.

## Files Changed

- tests/real_captures/*.sub (11 files, 192KB)
- tests/real_captures/manifest.json (11 test cases)
- scripts/benchmark_real.py (370 lines)
- REAL_TEST_RESULTS.md (benchmark results)
- docs/REAL_CAPTURE_ANALYSIS.md (comprehensive analysis)

🎯 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2026-02-15 18:08:48 -08:00

4.7 KiB
Raw Permalink Blame History

RF Device Identification - Real-World Benchmark Results

Date: 1771207572.2499428 Total Tests: 11 Source: Real Flipper Zero community captures

Overall Metrics

Top-K Accuracy

  • Top-1: 0.0%
  • Top-3: 0.0%
  • Top-5: 0.0%

Confidence Distribution

  • High (>80%): 1 (9.1%)
  • Medium (50-80%): 4 (36.4%)
  • Low (<50%): 1 (9.1%)
  • None found: 5 (45.5%)

Performance

  • Avg Parse Time: 0.28 ms
  • Avg Match Time: 128.19 ms
  • Total: 128.46 ms

By Category

Category Tests Top-1 Acc Top-3 Acc
doorbell 2 0.0% 0.0%
fan_controller 3 0.0% 0.0%
garage_door 2 0.0% 0.0%
led_controller 1 0.0% 0.0%
weather_sensor 3 0.0% 0.0%

Detailed Results

✗ lacrosse_tx141_real.sub

  • Expected: LaCrosse TX141-BV2
  • Category: weather_sensor
  • Got: Schou 72543 Day Rain Gauge, Motonet MTX Rain, MarQuant Rain Gauge, TFA Dostmann 30.3252.01/47.3006.01 Rain Gauge and Thermometer, ADE WS1907 (confidence: 69.1%)
  • Rank: Not Found

Top 5 Matches:

  1. Schou 72543 Day Rain Gauge, Motonet MTX Rain, MarQuant Rain Gauge, TFA Dostmann 30.3252.01/47.3006.01 Rain Gauge and Thermometer, ADE WS1907 (69.1%)
  2. Conrad S3318P, FreeTec NC-5849-913 temperature humidity sensor, ORIA WA50 ST389 temperature sensor (68.6%)
  3. WEC-2103 temperature/humidity sensor (68.6%)
  4. ThermoPro TX-2C Thermometer and Humidity sensor (68.0%)
  5. Biltema rain gauge (67.7%)

✗ acurite_02077m_real.sub

  • Expected: Acurite 02077M
  • Category: weather_sensor
  • Got: LaCrosse Technology View LTV-R1, LTV-R3 Rainfall Gauge, LTV-W1/W2 Wind Sensor (confidence: 76.5%)
  • Rank: Not Found

Top 5 Matches:

  1. LaCrosse Technology View LTV-R1, LTV-R3 Rainfall Gauge, LTV-W1/W2 Wind Sensor (76.5%)
  2. LaCrosse Technology View LTV-TH Thermo/Hygro Sensor (76.5%)
  3. LaCrosse TX35DTH-IT, TFA Dostmann 30.3155 Temperature/Humidity sensor (76.4%)
  4. TFA Marbella Pool Thermometer (76.4%)
  5. ThermoPro TP28b Super Long Range Wireless Meat Thermometer for Smoker BBQ Grill (76.4%)

✗ nexus_th_real.sub

  • Expected: Nexus Temperature-Humidity
  • Category: weather_sensor
  • Got: Oregon Scientific Weather Sensor (confidence: 76.9%)
  • Rank: 34

Top 5 Matches:

  1. Oregon Scientific Weather Sensor (76.9%)
  2. Oregon Scientific v3.0 (75.9%)
  3. Oregon Scientific v2.1 (74.2%)
  4. Ambient Weather F007TH (63.5%)
  5. Digitech XC-0324 / AmbientWeather FT005TH temp/hum sensor (58.4%)

✗ liftmaster_433_real.sub

  • Expected: LiftMaster
  • Category: garage_door
  • Got: AlectoV1 Weather Sensor (Alecto WS3500 WS4500 Ventus W155/W044 Oregon) (confidence: 49.4%)
  • Rank: Not Found

Top 5 Matches:

  1. AlectoV1 Weather Sensor (Alecto WS3500 WS4500 Ventus W155/W044 Oregon) (49.4%)
  2. Oregon Scientific SL109H Remote Thermal Hygro Sensor (49.4%)
  3. Philips outdoor temperature sensor (type AJ3650) (43.0%)
  4. Philips outdoor temperature sensor (type AJ7010) (43.0%)
  5. Globaltronics GT-WT-02 Sensor (39.5%)

✗ liftmaster_security2_raw_real.sub

  • Expected: LiftMaster Security+ 2.0
  • Category: garage_door
  • Got: No matches (confidence: 0.0%)
  • Rank: Not Found

✗ ge_doorbell_real.sub

  • Expected: GE Doorbell
  • Category: doorbell
  • Got: Conrad S3318P, FreeTec NC-5849-913 temperature humidity sensor, ORIA WA50 ST389 temperature sensor (confidence: 76.9%)
  • Rank: Not Found

Top 5 Matches:

  1. Conrad S3318P, FreeTec NC-5849-913 temperature humidity sensor, ORIA WA50 ST389 temperature sensor (76.9%)
  2. WEC-2103 temperature/humidity sensor (76.9%)
  3. ThermoPro TX-2C Thermometer and Humidity sensor (76.3%)
  4. Biltema rain gauge (75.8%)
  5. FT-004-B Temperature Sensor (75.7%)

✗ byron_doorbell_real.sub

  • Expected: Byron Doorbell
  • Category: doorbell
  • Got: Princeton (confidence: 92.9%)
  • Rank: Not Found

Top 5 Matches:

  1. Princeton (92.9%)
  2. Generic Remote SC226x EV1527 (84.1%)
  3. Elro DB286A Doorbell (83.2%)
  4. Blyss DC5-UK-WH (82.7%)
  5. Acurite 590TX Temperature with optional Humidity (82.7%)

✗ rgb_led_remote_real.sub

  • Expected: RGB LED Remote
  • Category: led_controller
  • Got: No matches (confidence: 0.0%)
  • Rank: Not Found

✗ ceiling_fan_real.sub

  • Expected: Ceiling Fan Remote
  • Category: fan_controller
  • Got: No matches (confidence: 0.0%)
  • Rank: Not Found

✗ ceiling_fan2_real.sub

  • Expected: Ceiling Fan Remote
  • Category: fan_controller
  • Got: No matches (confidence: 0.0%)
  • Rank: Not Found

✗ ceiling_fan_light_real.sub

  • Expected: Ceiling Fan Light
  • Category: fan_controller
  • Got: No matches (confidence: 0.0%)
  • Rank: Not Found
Reference in New Issue View Git Blame Copy Permalink