# Phase 1: RTL_433 Integration - Summary & Next Steps

**Date:** January 14, 2026
**Status:** ✅ Phase 1 Complete - Parser & Database Ready
**Next:** Phase 2 - Matcher Integration & Timing Analysis

---

## What Was Accomplished

### 1. Research & Analysis ✅
- **Comprehensive state-of-the-art research** documented in `docs/RF_SIGNAL_ANALYSIS_RESEARCH.md`
- Analyzed RTL_433 (260+ protocols), Flipper Zero (13k signals), URH, ML research
- Identified accuracy progression path: 60% → 95% in 6 phases
- Documented 7 device identification strategies

### 2. RTL_433 Repository Analysis ✅
- Cloned RTL_433 repository from GitHub
- Analyzed 255 device C source files
- Identified device registration structure (`r_device` structs)
- Found master device list (`rtl_433_devices.h` with 292 DECL macros)

### 3. Python Parser Development ✅
Created `scripts/parse_rtl433_devices.py` that:
- Parses C source code using regex patterns
- Extracts device metadata:
  - Device ID and human-readable name
  - Modulation type (OOK, FSK, etc.)
  - Timing parameters (pulse widths, gaps, resets)
  - Source file reference
- Infers device category from name
- Extracts manufacturer
- Exports to structured JSON

### 4. Protocol Database Generation ✅
Generated `data/rtl_433_protocols.json` with **286 device protocols**:

**By Category:**
- 116 Weather stations/sensors (40.6%)
- 52 Other devices (18.2%)
- 39 Generic sensors (13.6%)
- 25 TPMS tire pressure (8.7%)
- 23 Security devices (8.0%)
- 18 Home automation (6.3%)
- 7 Automotive (2.4%)
- 3 Energy meters (1.0%)
- 3 Lighting (1.0%)

**By Modulation:**
- 152 OOK (On-Off Keying) - 53.1%
- 105 FSK (Frequency Shift Keying) - 36.7%
- 19 OOK Manchester - 6.6%
- 5 FSK Manchester - 1.7%
- 5 Other modulations - 1.9%

**Top Manufacturers:**
1. Fine Offset - 15 devices
2. LaCrosse - 11 devices
3. Acurite - 8 devices
4. Bresser - 7 devices
5. TFA - 7 devices
6. ThermoPro - 6 devices
7. Auriol - 5 devices

### 5. Heatmap Visualization ✅
- Implemented Leaflet.heat plugin
- Confidence-based intensity coloring
- Frequency filtering support
- Smooth toggle between markers and heatmap
- Working and committed

---

## Example Parsed Devices

```json
{
  "device_id": "acurite_rain_896",
  "name": "Acurite 896 Rain Gauge",
  "modulation": "OOK",
  "short_width": 1000,
  "long_width": 2000,
  "gap_limit": 3500,
  "reset_limit": 5000,
  "category": "weather",
  "manufacturer": "Acurite",
  "source_file": "acurite.c"
}
```

```json
{
  "device_id": "tpms_ford",
  "name": "Ford TPMS",
  "modulation": "FSK",
  "short_width": 52,
  "long_width": 104,
  "gap_limit": 150,
  "reset_limit": 400,
  "category": "tpms",
  "manufacturer": "Ford",
  "source_file": "tpms_ford.c"
}
```

---

## Key Statistics

| Metric | Value |
|--------|-------|
| Total RTL_433 devices parsed | 286 |
| Weather/sensor devices | 155 (54%) |
| Automotive (TPMS + key fobs) | 32 (11%) |
| Home automation | 41 (14%) |
| OOK modulation devices | 152 (53%) |
| FSK modulation devices | 105 (37%) |
| Devices with timing data | 286 (100%) |
| Unique manufacturers | 50+ |

---

## What This Means for GigLez

### Current Accuracy: ~60-70%
Our simple matcher uses:
- 20 protocol patterns (manual)
- 50+ frequency-based patterns
- Basic modulation detection

### With RTL_433: Target ~75-80% (+15%)
We now have access to:
- **286 device protocols** (14x increase)
- Accurate timing parameters for matching
- Manufacturer and category data
- Comprehensive device names

### How the Improvement Works

**Before (Current):**
```
User uploads .sub file with Protocol: "Oregon"
→ Match against 20 manual patterns
→ Generic match: "Oregon Scientific Weather Station" (70% confidence)
```

**After (With RTL_433):**
```
User uploads .sub file with Protocol: "Oregon"
→ Match against 286 RTL_433 protocols
→ Find "oregon_scientific" in database
→ Specific match: "Oregon Scientific Weather Sensor" (85% confidence)
→ Category: "weather", Manufacturer: "Oregon"
```

**Even Better (RAW captures with timing):**
```
User uploads RAW capture (Protocol: "RAW")
→ Extract pulse timings: short=500µs, long=1000µs, gap=3000µs
→ Match against RTL_433 timing signatures
→ Find best match: "Fine Offset WH25" (75% confidence)
→ Previously would have been "Unknown Device"
```

---

## Next Steps

### Phase 2: Matcher Integration (In Progress)

#### Task 2.1: Enhanced Matcher with RTL_433 Loader
**File:** `src/matcher/rtl433_matcher.py`

Create new matcher class that:
1. Loads `data/rtl_433_protocols.json` on startup
2. Builds searchable index by:
   - Device ID
   - Device name keywords
   - Modulation type
   - Category
3. Implements matching strategies:
   - Exact device ID match
   - Fuzzy name match
   - Modulation + timing match
   - Category-based fallback

**Example structure:**
```python
class RTL433Matcher:
    def __init__(self, json_path):
        with open(json_path) as f:
            data = json.load(f)
            self.devices = data['devices']
        self.build_indexes()

    def match(self, protocol, modulation, timing_params):
        # Try exact protocol match
        # Try timing-based match
        # Try modulation + category
        return matches
```

#### Task 2.2: Integrate into Main Matcher
**File:** `src/matcher/simple_matcher.py` (enhance existing)

Add RTL_433 as new matching strategy:
```python
def match(self, frequency, protocol, preset):
    matches = []

    # NEW: RTL_433 protocol matching (highest confidence)
    if protocol != "RAW":
        rtl_matches = self.rtl433_matcher.match_by_protocol(protocol)
        matches.extend([(m, 0.85, "rtl433_protocol") for m in rtl_matches])

    # Existing: Protocol matching (high confidence)
    if protocol and protocol != "RAW":
        protocol_matches = self._match_by_protocol(protocol)
        matches.extend([(m, 0.70, "protocol") for m in protocol_matches])

    # ... rest of matching logic
```

#### Task 2.3: Update API Upload Endpoint
**File:** `src/api/main_simple.py`

No changes needed! Matcher is already integrated in upload pipeline.
The enhanced matcher will automatically be used.

#### Task 2.4: Test with Real Captures
1. Re-match existing 20 captures with new matcher
2. Upload new test captures
3. Compare accuracy: old vs. new
4. Document improvement percentage

**Expected Result:**
- Acurite captures: 70% → 85% confidence
- Oregon captures: 70% → 90% confidence
- Generic 433MHz: 40% → 60% confidence

---

### Phase 3: Timing Analysis (Next)

#### Task 3.1: RAW Data Parser
**File:** `src/parser/raw_parser.py`

Parse `RAW_Data` field from .sub files:
```python
def parse_raw_data(raw_data_string):
    """
    Input: "2980 -240 520 -980 520 -980 ..."
    Output: {
        'pulses': [2980, 240, 520, 980, ...],
        'high_pulses': [2980, 520, 520, ...],
        'low_pulses': [240, 980, 980, ...],
        'mean_high': 840,
        'mean_low': 733,
        'ratio': 1.15,
        'encoding': 'PWM'  # or 'PPM', 'Manchester', etc.
    }
    """
```

#### Task 3.2: Timing Matcher
**File:** `src/matcher/timing_matcher.py`

Match RAW captures against RTL_433 timing signatures:
```python
def match_timing(raw_data, rtl433_database):
    """
    Compare extracted timing against known patterns
    Tolerance: ±15% for pulse widths
    """
    parsed = parse_raw_data(raw_data)

    matches = []
    for device in rtl433_database:
        if device.short_width:
            similarity = calculate_timing_similarity(
                parsed,
                device.short_width,
                device.long_width,
                tolerance=0.15
            )
            if similarity > 0.6:
                matches.append((device, similarity))

    return sorted(matches, key=lambda x: x[1], reverse=True)
```

#### Task 3.3: Integration
Add timing analysis to main matcher:
```python
# In enhanced matcher
if capture.raw_data:
    timing_matches = timing_matcher.match_timing(
        capture.raw_data,
        self.rtl433_devices
    )
    matches.extend([(m, conf*0.75, "timing") for m, conf in timing_matches])
```

**Expected Accuracy:** 80-85% overall (+5-10% for RAW captures)

---

### Phase 4: ML Infrastructure Planning (Future)

#### Task 4.1: Dataset Collection System
- Track all user uploads with GPS coords
- Store verified device identifications
- Build training dataset (target: 10,000+ captures)

#### Task 4.2: Feature Engineering
- Timing features (mean, std, ratio, entropy)
- Spectral features (if I/Q data available)
- Statistical features (autocorrelation, etc.)

#### Task 4.3: Model Selection & Training
Options to evaluate:
1. **SVM with RF features** (simplest, 90-92% accuracy)
2. **Random Forest** (moderate, 92-94% accuracy)
3. **CNN on spectrograms** (complex, 94-96% accuracy)
4. **LSTM on pulse sequences** (complex, 93-95% accuracy)

#### Task 4.4: Deployment Strategy
- Train on GigLez server or cloud GPU
- Export model to ONNX for fast inference
- Deploy API endpoint for classification
- Fallback to rule-based if ML unavailable

**Expected Accuracy:** 90-95% (state-of-the-art)

---

## Files Created/Modified

### New Files:
- `docs/RF_SIGNAL_ANALYSIS_RESEARCH.md` - Comprehensive research doc (1,100+ lines)
- `scripts/parse_rtl433_devices.py` - RTL_433 parser (350 lines)
- `data/rtl_433_protocols.json` - Device database (4,500 lines, 286 devices)
- `static/js/detail-modal.js` - Enhanced (with debug logging)
- `templates/index.html` - Added Leaflet.heat plugin, fixed modal
- `static/js/map.js` - Added heatmap rendering

### Modified Files:
- Heatmap implementation in map.js
- Modal fixes in detail-modal.js and index.html

---

## Metrics & Goals

### Current State (Before RTL_433):
- **Accuracy:** ~60-70%
- **Known protocols:** 20
- **Device database size:** ~50 entries
- **RAW capture handling:** Poor (generic matches only)

### Phase 1 Complete (Parser Ready):
- **RTL_433 protocols parsed:** 286
- **Database generated:** ✅ Yes
- **Parser tested:** ✅ Yes
- **Integration status:** 🔄 Next step

### Phase 2 Target (After Integration):
- **Accuracy:** ~75-80% (+15%)
- **Known protocols:** 286 (14x increase)
- **RAW capture handling:** Improved with timing
- **Timeline:** 1-2 weeks

### Phase 3 Target (Timing Analysis):
- **Accuracy:** ~80-85% (+5%)
- **RAW captures:** 75%+ identification rate
- **Timeline:** 2-3 weeks

### Phase 4+ Target (ML):
- **Accuracy:** 90-95% (state-of-the-art)
- **Timeline:** 3-6 months (requires dataset collection)

---

## Commands to Run

### Re-parse RTL_433 (if needed):
```bash
python3 scripts/parse_rtl433_devices.py /tmp/rtl_433/
```

### View database summary:
```bash
jq '.total_devices, .devices | group_by(.category) | map({category: .[0].category, count: length})' data/rtl_433_protocols.json
```

### Count by manufacturer:
```bash
jq '[.devices[].manufacturer] | group_by(.) | map({manufacturer: .[0], count: length}) | sort_by(-.count) | .[0:10]' data/rtl_433_protocols.json
```

---

## References

- **RTL_433 Repository:** https://github.com/merbanan/rtl_433
- **RTL_433 Documentation:** https://github.com/merbanan/rtl_433/tree/master/docs
- **Device List:** `rtl_433/include/rtl_433_devices.h`
- **Device Sources:** `rtl_433/src/devices/*.c` (255 files)

---

## Success Criteria

### Phase 1: ✅ COMPLETE
- [x] Parse RTL_433 C source code
- [x] Extract 250+ device protocols
- [x] Generate structured JSON database
- [x] Document all findings
- [x] Commit and push

### Phase 2: 🔄 IN PROGRESS
- [ ] Create RTL_433 matcher class
- [ ] Integrate into existing matcher
- [ ] Test with sample captures
- [ ] Measure accuracy improvement
- [ ] Document results

### Phase 3: ⏳ PENDING
- [ ] Implement RAW data parser
- [ ] Create timing matcher
- [ ] Integrate timing analysis
- [ ] Test with RAW captures
- [ ] Target 80-85% accuracy

---

**Status:** Phase 1 complete! Ready to proceed with Phase 2 integration.

**Estimated Impact:** +15-20% accuracy improvement once integrated.

**Next Action:** Create `src/matcher/rtl433_matcher.py` and integrate into main matcher.