giglez/docs/REAL_CAPTURE_ANALYSIS.md

Real-World Capture Analysis - Gap Between Synthetic and Real Performance

Date: 2026-02-15 Benchmark: Real Flipper Zero Community Captures vs Synthetic Test Signals

Executive Summary

CRITICAL FINDING: The RF device identification system achieves 0% accuracy on real-world captures vs 33% on synthetic signals. This represents a complete failure to generalize from synthetic to real-world data.

Performance Comparison

Metric	Synthetic	Real-World	Delta
Top-1 Accuracy	33.3% (4/12)	0.0% (0/11)	-33.3%
Top-3 Accuracy	50.0% (6/12)	0.0% (0/11)	-50.0%
High Confidence (>80%)	66.7%	9.1%	-57.6%
No Matches Found	8.3% (1/12)	45.5% (5/11)	+37.2%

Root Causes

1. Decoded vs RAW File Format (5/11 failures = 45%)

Problem: Many real Flipper captures are already decoded, not RAW.

Evidence:

• All 3 ceiling fan files: Protocol: Holtek_HT12X (decoded)
• RGB LED remote: Already decoded
• LiftMaster Security+ 2.0: Already decoded

Example (ceiling_fan_real.sub):

Filetype: Flipper SubGhz Key File
Version: 1
Frequency: 315000000
Preset: FuriHalSubGhzPresetOok650Async
Protocol: Holtek_HT12X  # <-- Already decoded!
Bit: 12
Key: 00 00 00 00 00 00 00 BF
TE: 470

Impact: Our system only processes RAW_Data. Decoded files have has_raw_data=False, so the pattern decoder immediately returns empty results.

Fix Required: Add support for decoded .sub files:

1. Parse Protocol, Bit, Key, TE fields
2. Match against database by protocol name + timing element
3. Synthesize timing from TE (timing element) if needed

---

2. Real Signal Noise & Complexity (3/11 failures = 27%)

Problem: Real captures have:

• Variable signal quality
• Multi-packet transmissions
• Long pulse trains (LaCrosse: 131 pulses total)
• Interference and jitter beyond our 15% tolerance

Evidence:

• lacrosse_tx141_real.sub: 131 pulses, very long RAW_Data
• Top match: "Schou 72543 Day Rain Gauge" (69.1%) - WRONG
• Expected "LaCrosse TX141-BV2" not found in top 100

Analysis: Real LaCrosse signals are much longer and noisier than our 40-bit synthetic version.

---

3. Missing Protocols in Database (3/11 failures = 27%)

Problem: Expected protocols not in our 299-protocol database:

• "Acurite 02077M" → Not in database (we have Acurite 609TXC, but not 02077M variant)
• "GE Doorbell 19297" → Not in database
• "Byron Doorbell DB421E" → Not in database (Byron got Princeton at 92.9%)

Evidence:

Byron doorbell → Princeton (92.9% confidence)
GE doorbell → Conrad S3318P sensor (76.9% confidence)

Impact: Even with perfect timing extraction, we can't match protocols that don't exist in the database.

---

4. Protocol Name Matching Too Strict (1/11 = 9%)

Problem: nexus_th_real.sub matched "Nexus Temperature-Humidity" but at rank 34.

Top 5 Results:

1. Oregon Scientific Weather Sensor (76.9%)
2. Oregon Scientific v3.0 (75.9%)
3. Oregon Scientific v2.1 (74.2%)
4. Ambient Weather F007TH (63.5%)
5. Digitech XC-0324 (58.4%) ...
6. Nexus Temperature-Humidity

Analysis: Nexus is getting matched, but Oregon protocols score higher. This suggests:

• Timing similarity between Nexus and Oregon
• Preamble detection favoring Oregon
• Need protocol-family grouping (Nexus vs Oregon shouldn't compete directly)

---

Detailed Failure Analysis

Weather Sensors (0/3 correct)

1. LaCrosse TX141-BV2 Real Capture

• Expected: LaCrosse TX141-BV2
• Got: Schou 72543 Day Rain Gauge (69.1%)
• Rank: Not found

Issue: Real capture has 131 pulses vs synthetic 40 bits. Likely multiple transmissions in one capture.

2. Acurite 02077M

• Expected: Acurite 02077M
• Got: LaCrosse Technology View LTV-R1 (76.5%)
• Rank: Not found

Issue: Acurite 02077M not in database. We have Acurite 609TXC but not this model.

3. Nexus Temperature-Humidity

• Expected: Nexus Temperature-Humidity
• Got: Oregon Scientific Weather Sensor (76.9%)
• Rank: 34

Issue: Protocol exists but ranks low. Oregon and Nexus are similar protocols competing.

---

Garage Doors (0/2 correct)

1. LiftMaster 433MHz

• Expected: LiftMaster
• Got: AlectoV1 Weather Sensor (49.4%)
• Rank: Not found

Issue: LiftMaster not in database. Low confidence (49%) suggests poor timing match.

2. LiftMaster Security+ 2.0

• Expected: LiftMaster Security+ 2.0
• Got: No matches (0%)
• Rank: Not found

Issue: Decoded file format. No RAW_Data to process. Security+ 2.0 uses rolling codes.

---

Doorbells (0/2 correct)

1. GE Doorbell 19297

• Expected: GE Doorbell
• Got: Conrad S3318P sensor (76.9%)
• Rank: Not found

Issue: GE Doorbell not in database. Misidentified as temperature sensor.

2. Byron Doorbell DB421E

• Expected: Byron Doorbell
• Got: Princeton (92.9%)
• Rank: Not found

Issue: Byron not in database. High confidence match to Princeton suggests similar timing (Princeton-like protocol).

---

LED/Fan Controllers (0/4 correct)

All 4 files: No matches (0%)

Issue: All are decoded format, not RAW. Our system cannot process them.

Files:

• rgb_led_remote_real.sub - Decoded
• ceiling_fan_real.sub - Decoded (Holtek_HT12X)
• ceiling_fan2_real.sub - Decoded (Holtek_HT12X)
• ceiling_fan_light_real.sub - Decoded (Holtek_HT12X)

---

Recommendations

Priority 1: Support Decoded .sub Files (Fixes 45% of failures)

Implementation:

def parse_sub_file(filepath):
    """Parse both RAW and decoded .sub files"""
    metadata = SignalMetadata()

    with open(filepath) as f:
        # ... existing parsing ...

        # If no RAW_Data but has Protocol field
        if not metadata.has_raw_data and metadata.protocol:
            # This is a decoded file
            # Extract: Protocol, Bit, Key, TE
            # Match by protocol name + TE timing
            metadata.is_decoded = True
            metadata.timing_element = te_value

    return metadata

Database Matching:

• For decoded files, match primarily by Protocol field name
• Use TE (timing element) as secondary confirmation
• Skip RAW pattern analysis

---

Priority 2: Add Missing Protocols (Fixes 27% of failures)

Protocols to Add:

1. Acurite 02077M - Weather station (433.92 MHz)
2. GE Doorbell 19297 - Doorbell (433.92 MHz)
3. Byron DB421E - Doorbell (433.92 MHz)
4. LiftMaster (generic 433 MHz) - Garage door
5. Holtek HT12X - LED/Fan remote encoder chip

Source: Import from real captures or RTL_433 database updates.

---

Priority 3: Improve Real Signal Robustness (Fixes 27% of failures)

Issues:

• Long multi-packet captures (131 pulses vs 40 bits)
• Variable signal quality
• Packet repetition detection needed

Fixes:

1. Packet Segmentation: Split long RAW_Data into individual transmissions
2. Repetition Detection: Find repeating patterns, score best match
3. Noise Filtering: Increase jitter tolerance from 15% to 25% for real data
4. Confidence Calibration: Lower thresholds for real-world data

---

Priority 4: Protocol Family Grouping (Fixes 9% of failures)

Issue: Nexus ranks #34, beaten by Oregon Scientific variants.

Fix:

• Group similar protocols (Nexus, Oregon Scientific, Acurite all use similar encoding)
• Score by protocol family first, then by specific model
• Boost exact name matches over family matches

---

Performance Expectations After Fixes

Fix	Impact	Expected Accuracy
Support decoded files	+45% (5/11)	45%
Add missing protocols	+27% (3/11)	72%
Improve robustness	+18% (2/11)	90%
Family grouping	+9% (1/11)	100%

Realistic Target: 60-70% top-1 accuracy after implementing Priority 1 and 2.

---

Synthetic vs Real: Why the Gap?

Synthetic Signals (33% accuracy)

• ✅ Perfect timing (no jitter beyond 5-15%)
• ✅ Single packet per file
• ✅ Known protocol parameters
• ✅ All RAW format
• ❌ Unrealistic simplicity

Real Captures (0% accuracy)

• ❌ 45% are decoded (not RAW)
• ❌ 27% use protocols not in database
• ❌ Variable signal quality
• ❌ Multi-packet transmissions
• ❌ Real-world interference
• ✅ Actual device behavior

Conclusion: Our system was optimized for synthetic signals that don't represent real-world usage.

---

Next Steps

1. Immediate: Implement decoded .sub file support (Priority 1)
2. Short-term: Add missing protocols from real captures (Priority 2)
3. Medium-term: Improve noise tolerance and packet segmentation (Priority 3)
4. Long-term: Collect more real captures for training/validation

---

Generated: 2026-02-15 Benchmark Version: Iteration 6 (post-precision tuning) Database Size: 299 protocols (RTL_433 imported) Real Captures Source: UberGuidoZ/Flipper repository