giglez/docs/REAL_CAPTURE_ANALYSIS.md

# Real-World Capture Analysis - Gap Between Synthetic and Real Performance

**Date**: 2026-02-15
**Benchmark**: Real Flipper Zero Community Captures vs Synthetic Test Signals

## Executive Summary

**CRITICAL FINDING**: The RF device identification system achieves **0% accuracy on real-world captures** vs **33% on synthetic signals**. This represents a complete failure to generalize from synthetic to real-world data.

## Performance Comparison

| Metric | Synthetic | Real-World | Delta |
|--------|-----------|------------|-------|
| Top-1 Accuracy | 33.3% (4/12) | **0.0%** (0/11) | **-33.3%** |
| Top-3 Accuracy | 50.0% (6/12) | **0.0%** (0/11) | **-50.0%** |
| High Confidence (>80%) | 66.7% | **9.1%** | -57.6% |
| No Matches Found | 8.3% (1/12) | **45.5%** (5/11) | +37.2% |

## Root Causes

### 1. **Decoded vs RAW File Format** (5/11 failures = 45%)

**Problem**: Many real Flipper captures are **already decoded**, not RAW.

**Evidence**:
- All 3 ceiling fan files: `Protocol: Holtek_HT12X` (decoded)
- RGB LED remote: Already decoded
- LiftMaster Security+ 2.0: Already decoded

**Example** (`ceiling_fan_real.sub`):
```
Filetype: Flipper SubGhz Key File
Version: 1
Frequency: 315000000
Preset: FuriHalSubGhzPresetOok650Async
Protocol: Holtek_HT12X  # <-- Already decoded!
Bit: 12
Key: 00 00 00 00 00 00 00 BF
TE: 470
```

**Impact**: Our system **only processes RAW_Data**. Decoded files have `has_raw_data=False`, so the pattern decoder immediately returns empty results.

**Fix Required**: Add support for decoded .sub files:
1. Parse `Protocol`, `Bit`, `Key`, `TE` fields
2. Match against database by protocol name + timing element
3. Synthesize timing from TE (timing element) if needed

---

### 2. **Real Signal Noise & Complexity** (3/11 failures = 27%)

**Problem**: Real captures have:
- Variable signal quality
- Multi-packet transmissions
- Long pulse trains (LaCrosse: 131 pulses total)
- Interference and jitter beyond our 15% tolerance

**Evidence**:
- `lacrosse_tx141_real.sub`: 131 pulses, very long RAW_Data
- Top match: "Schou 72543 Day Rain Gauge" (69.1%) - WRONG
- Expected "LaCrosse TX141-BV2" not found in top 100

**Analysis**: Real LaCrosse signals are much longer and noisier than our 40-bit synthetic version.

---

### 3. **Missing Protocols in Database** (3/11 failures = 27%)

**Problem**: Expected protocols not in our 299-protocol database:
- "Acurite 02077M" → Not in database (we have Acurite 609TXC, but not 02077M variant)
- "GE Doorbell 19297" → Not in database
- "Byron Doorbell DB421E" → Not in database (Byron got Princeton at 92.9%)

**Evidence**:
```
Byron doorbell → Princeton (92.9% confidence)
GE doorbell → Conrad S3318P sensor (76.9% confidence)
```

**Impact**: Even with perfect timing extraction, we can't match protocols that don't exist in the database.

---

### 4. **Protocol Name Matching Too Strict** (1/11 = 9%)

**Problem**: `nexus_th_real.sub` matched "Nexus Temperature-Humidity" but at **rank 34**.

**Top 5 Results**:
1. Oregon Scientific Weather Sensor (76.9%)
2. Oregon Scientific v3.0 (75.9%)
3. Oregon Scientific v2.1 (74.2%)
4. Ambient Weather F007TH (63.5%)
5. Digitech XC-0324 (58.4%)
...
34. Nexus Temperature-Humidity

**Analysis**: Nexus is getting matched, but Oregon protocols score higher. This suggests:
- Timing similarity between Nexus and Oregon
- Preamble detection favoring Oregon
- Need protocol-family grouping (Nexus vs Oregon shouldn't compete directly)

---

## Detailed Failure Analysis

### Weather Sensors (0/3 correct)

#### 1. LaCrosse TX141-BV2 Real Capture
- **Expected**: LaCrosse TX141-BV2
- **Got**: Schou 72543 Day Rain Gauge (69.1%)
- **Rank**: Not found

**Issue**: Real capture has 131 pulses vs synthetic 40 bits. Likely multiple transmissions in one capture.

#### 2. Acurite 02077M
- **Expected**: Acurite 02077M
- **Got**: LaCrosse Technology View LTV-R1 (76.5%)
- **Rank**: Not found

**Issue**: Acurite 02077M not in database. We have Acurite 609TXC but not this model.

#### 3. Nexus Temperature-Humidity
- **Expected**: Nexus Temperature-Humidity
- **Got**: Oregon Scientific Weather Sensor (76.9%)
- **Rank**: 34

**Issue**: Protocol exists but ranks low. Oregon and Nexus are similar protocols competing.

---

### Garage Doors (0/2 correct)

#### 1. LiftMaster 433MHz
- **Expected**: LiftMaster
- **Got**: AlectoV1 Weather Sensor (49.4%)
- **Rank**: Not found

**Issue**: LiftMaster not in database. Low confidence (49%) suggests poor timing match.

#### 2. LiftMaster Security+ 2.0
- **Expected**: LiftMaster Security+ 2.0
- **Got**: No matches (0%)
- **Rank**: Not found

**Issue**: **Decoded file format**. No RAW_Data to process. Security+ 2.0 uses rolling codes.

---

### Doorbells (0/2 correct)

#### 1. GE Doorbell 19297
- **Expected**: GE Doorbell
- **Got**: Conrad S3318P sensor (76.9%)
- **Rank**: Not found

**Issue**: GE Doorbell not in database. Misidentified as temperature sensor.

#### 2. Byron Doorbell DB421E
- **Expected**: Byron Doorbell
- **Got**: Princeton (92.9%)
- **Rank**: Not found

**Issue**: Byron not in database. High confidence match to Princeton suggests similar timing (Princeton-like protocol).

---

### LED/Fan Controllers (0/4 correct)

#### All 4 files: No matches (0%)

**Issue**: **All are decoded format**, not RAW. Our system cannot process them.

**Files**:
- `rgb_led_remote_real.sub` - Decoded
- `ceiling_fan_real.sub` - Decoded (Holtek_HT12X)
- `ceiling_fan2_real.sub` - Decoded (Holtek_HT12X)
- `ceiling_fan_light_real.sub` - Decoded (Holtek_HT12X)

---

## Recommendations

### Priority 1: Support Decoded .sub Files (Fixes 45% of failures)

**Implementation**:
```python
def parse_sub_file(filepath):
    """Parse both RAW and decoded .sub files"""
    metadata = SignalMetadata()

    with open(filepath) as f:
        # ... existing parsing ...

        # If no RAW_Data but has Protocol field
        if not metadata.has_raw_data and metadata.protocol:
            # This is a decoded file
            # Extract: Protocol, Bit, Key, TE
            # Match by protocol name + TE timing
            metadata.is_decoded = True
            metadata.timing_element = te_value

    return metadata
```

**Database Matching**:
- For decoded files, match primarily by `Protocol` field name
- Use `TE` (timing element) as secondary confirmation
- Skip RAW pattern analysis

---

### Priority 2: Add Missing Protocols (Fixes 27% of failures)

**Protocols to Add**:
1. **Acurite 02077M** - Weather station (433.92 MHz)
2. **GE Doorbell 19297** - Doorbell (433.92 MHz)
3. **Byron DB421E** - Doorbell (433.92 MHz)
4. **LiftMaster** (generic 433 MHz) - Garage door
5. **Holtek HT12X** - LED/Fan remote encoder chip

**Source**: Import from real captures or RTL_433 database updates.

---

### Priority 3: Improve Real Signal Robustness (Fixes 27% of failures)

**Issues**:
- Long multi-packet captures (131 pulses vs 40 bits)
- Variable signal quality
- Packet repetition detection needed

**Fixes**:
1. **Packet Segmentation**: Split long RAW_Data into individual transmissions
2. **Repetition Detection**: Find repeating patterns, score best match
3. **Noise Filtering**: Increase jitter tolerance from 15% to 25% for real data
4. **Confidence Calibration**: Lower thresholds for real-world data

---

### Priority 4: Protocol Family Grouping (Fixes 9% of failures)

**Issue**: Nexus ranks #34, beaten by Oregon Scientific variants.

**Fix**:
- Group similar protocols (Nexus, Oregon Scientific, Acurite all use similar encoding)
- Score by protocol family first, then by specific model
- Boost exact name matches over family matches

---

## Performance Expectations After Fixes

| Fix | Impact | Expected Accuracy |
|-----|--------|-------------------|
| Support decoded files | +45% (5/11) | 45% |
| Add missing protocols | +27% (3/11) | 72% |
| Improve robustness | +18% (2/11) | 90% |
| Family grouping | +9% (1/11) | 100% |

**Realistic Target**: 60-70% top-1 accuracy after implementing Priority 1 and 2.

---

## Synthetic vs Real: Why the Gap?

### Synthetic Signals (33% accuracy)
- ✅ Perfect timing (no jitter beyond 5-15%)
- ✅ Single packet per file
- ✅ Known protocol parameters
- ✅ All RAW format
- ❌ Unrealistic simplicity

### Real Captures (0% accuracy)
- ❌ 45% are decoded (not RAW)
- ❌ 27% use protocols not in database
- ❌ Variable signal quality
- ❌ Multi-packet transmissions
- ❌ Real-world interference
- ✅ Actual device behavior

**Conclusion**: Our system was optimized for synthetic signals that don't represent real-world usage.

---

## Next Steps

1. **Immediate**: Implement decoded .sub file support (Priority 1)
2. **Short-term**: Add missing protocols from real captures (Priority 2)
3. **Medium-term**: Improve noise tolerance and packet segmentation (Priority 3)
4. **Long-term**: Collect more real captures for training/validation

---

**Generated**: 2026-02-15
**Benchmark Version**: Iteration 6 (post-precision tuning)
**Database Size**: 299 protocols (RTL_433 imported)
**Real Captures Source**: UberGuidoZ/Flipper repository