giglez/docs/PHASE1_RTL433_INTEGRATION_SUMMARY.md

Phase 1: RTL_433 Integration - Summary & Next Steps

Date: January 14, 2026 Status: ✅ Phase 1 Complete - Parser & Database Ready Next: Phase 2 - Matcher Integration & Timing Analysis

---

What Was Accomplished

1. Research & Analysis ✅

• Comprehensive state-of-the-art research documented in docs/RF_SIGNAL_ANALYSIS_RESEARCH.md
• Analyzed RTL_433 (260+ protocols), Flipper Zero (13k signals), URH, ML research
• Identified accuracy progression path: 60% → 95% in 6 phases
• Documented 7 device identification strategies

2. RTL_433 Repository Analysis ✅

• Cloned RTL_433 repository from GitHub
• Analyzed 255 device C source files
• Identified device registration structure (r_device structs)
• Found master device list (rtl_433_devices.h with 292 DECL macros)

3. Python Parser Development ✅

Created scripts/parse_rtl433_devices.py that:

• Parses C source code using regex patterns
• Extracts device metadata:
  • Device ID and human-readable name
  • Modulation type (OOK, FSK, etc.)
  • Timing parameters (pulse widths, gaps, resets)
  • Source file reference
• Infers device category from name
• Extracts manufacturer
• Exports to structured JSON

4. Protocol Database Generation ✅

Generated data/rtl_433_protocols.json with 286 device protocols:

By Category:

• 116 Weather stations/sensors (40.6%)
• 52 Other devices (18.2%)
• 39 Generic sensors (13.6%)
• 25 TPMS tire pressure (8.7%)
• 23 Security devices (8.0%)
• 18 Home automation (6.3%)
• 7 Automotive (2.4%)
• 3 Energy meters (1.0%)
• 3 Lighting (1.0%)

By Modulation:

• 152 OOK (On-Off Keying) - 53.1%
• 105 FSK (Frequency Shift Keying) - 36.7%
• 19 OOK Manchester - 6.6%
• 5 FSK Manchester - 1.7%
• 5 Other modulations - 1.9%

Top Manufacturers:

1. Fine Offset - 15 devices
2. LaCrosse - 11 devices
3. Acurite - 8 devices
4. Bresser - 7 devices
5. TFA - 7 devices
6. ThermoPro - 6 devices
7. Auriol - 5 devices

5. Heatmap Visualization ✅

• Implemented Leaflet.heat plugin
• Confidence-based intensity coloring
• Frequency filtering support
• Smooth toggle between markers and heatmap
• Working and committed

---

Example Parsed Devices

{
  "device_id": "acurite_rain_896",
  "name": "Acurite 896 Rain Gauge",
  "modulation": "OOK",
  "short_width": 1000,
  "long_width": 2000,
  "gap_limit": 3500,
  "reset_limit": 5000,
  "category": "weather",
  "manufacturer": "Acurite",
  "source_file": "acurite.c"
}

{
  "device_id": "tpms_ford",
  "name": "Ford TPMS",
  "modulation": "FSK",
  "short_width": 52,
  "long_width": 104,
  "gap_limit": 150,
  "reset_limit": 400,
  "category": "tpms",
  "manufacturer": "Ford",
  "source_file": "tpms_ford.c"
}

---

Key Statistics

Metric	Value
Total RTL_433 devices parsed	286
Weather/sensor devices	155 (54%)
Automotive (TPMS + key fobs)	32 (11%)
Home automation	41 (14%)
OOK modulation devices	152 (53%)
FSK modulation devices	105 (37%)
Devices with timing data	286 (100%)
Unique manufacturers	50+

---

What This Means for GigLez

Current Accuracy: ~60-70%

Our simple matcher uses:

• 20 protocol patterns (manual)
• 50+ frequency-based patterns
• Basic modulation detection

With RTL_433: Target ~75-80% (+15%)

We now have access to:

• 286 device protocols (14x increase)
• Accurate timing parameters for matching
• Manufacturer and category data
• Comprehensive device names

How the Improvement Works

Before (Current):

User uploads .sub file with Protocol: "Oregon"
→ Match against 20 manual patterns
→ Generic match: "Oregon Scientific Weather Station" (70% confidence)

After (With RTL_433):

User uploads .sub file with Protocol: "Oregon"
→ Match against 286 RTL_433 protocols
→ Find "oregon_scientific" in database
→ Specific match: "Oregon Scientific Weather Sensor" (85% confidence)
→ Category: "weather", Manufacturer: "Oregon"

Even Better (RAW captures with timing):

User uploads RAW capture (Protocol: "RAW")
→ Extract pulse timings: short=500µs, long=1000µs, gap=3000µs
→ Match against RTL_433 timing signatures
→ Find best match: "Fine Offset WH25" (75% confidence)
→ Previously would have been "Unknown Device"

---

Next Steps

Phase 2: Matcher Integration (In Progress)

Task 2.1: Enhanced Matcher with RTL_433 Loader

File: src/matcher/rtl433_matcher.py

Create new matcher class that:

1. Loads data/rtl_433_protocols.json on startup
2. Builds searchable index by:
   • Device ID
   • Device name keywords
   • Modulation type
   • Category
3. Implements matching strategies:
   • Exact device ID match
   • Fuzzy name match
   • Modulation + timing match
   • Category-based fallback

Example structure:

class RTL433Matcher:
    def __init__(self, json_path):
        with open(json_path) as f:
            data = json.load(f)
            self.devices = data['devices']
        self.build_indexes()

    def match(self, protocol, modulation, timing_params):
        # Try exact protocol match
        # Try timing-based match
        # Try modulation + category
        return matches

Task 2.2: Integrate into Main Matcher

File: src/matcher/simple_matcher.py (enhance existing)

Add RTL_433 as new matching strategy:

def match(self, frequency, protocol, preset):
    matches = []

    # NEW: RTL_433 protocol matching (highest confidence)
    if protocol != "RAW":
        rtl_matches = self.rtl433_matcher.match_by_protocol(protocol)
        matches.extend([(m, 0.85, "rtl433_protocol") for m in rtl_matches])

    # Existing: Protocol matching (high confidence)
    if protocol and protocol != "RAW":
        protocol_matches = self._match_by_protocol(protocol)
        matches.extend([(m, 0.70, "protocol") for m in protocol_matches])

    # ... rest of matching logic

Task 2.3: Update API Upload Endpoint

File: src/api/main_simple.py

No changes needed! Matcher is already integrated in upload pipeline. The enhanced matcher will automatically be used.

Task 2.4: Test with Real Captures

1. Re-match existing 20 captures with new matcher
2. Upload new test captures
3. Compare accuracy: old vs. new
4. Document improvement percentage

Expected Result:

• Acurite captures: 70% → 85% confidence
• Oregon captures: 70% → 90% confidence
• Generic 433MHz: 40% → 60% confidence

---

Phase 3: Timing Analysis (Next)

Task 3.1: RAW Data Parser

File: src/parser/raw_parser.py

Parse RAW_Data field from .sub files:

def parse_raw_data(raw_data_string):
    """
    Input: "2980 -240 520 -980 520 -980 ..."
    Output: {
        'pulses': [2980, 240, 520, 980, ...],
        'high_pulses': [2980, 520, 520, ...],
        'low_pulses': [240, 980, 980, ...],
        'mean_high': 840,
        'mean_low': 733,
        'ratio': 1.15,
        'encoding': 'PWM'  # or 'PPM', 'Manchester', etc.
    }
    """

Task 3.2: Timing Matcher

File: src/matcher/timing_matcher.py

Match RAW captures against RTL_433 timing signatures:

def match_timing(raw_data, rtl433_database):
    """
    Compare extracted timing against known patterns
    Tolerance: ±15% for pulse widths
    """
    parsed = parse_raw_data(raw_data)

    matches = []
    for device in rtl433_database:
        if device.short_width:
            similarity = calculate_timing_similarity(
                parsed,
                device.short_width,
                device.long_width,
                tolerance=0.15
            )
            if similarity > 0.6:
                matches.append((device, similarity))

    return sorted(matches, key=lambda x: x[1], reverse=True)

Task 3.3: Integration

Add timing analysis to main matcher:

# In enhanced matcher
if capture.raw_data:
    timing_matches = timing_matcher.match_timing(
        capture.raw_data,
        self.rtl433_devices
    )
    matches.extend([(m, conf*0.75, "timing") for m, conf in timing_matches])

Expected Accuracy: 80-85% overall (+5-10% for RAW captures)

---

Phase 4: ML Infrastructure Planning (Future)

Task 4.1: Dataset Collection System

• Track all user uploads with GPS coords
• Store verified device identifications
• Build training dataset (target: 10,000+ captures)

Task 4.2: Feature Engineering

• Timing features (mean, std, ratio, entropy)
• Spectral features (if I/Q data available)
• Statistical features (autocorrelation, etc.)

Task 4.3: Model Selection & Training

Options to evaluate:

1. SVM with RF features (simplest, 90-92% accuracy)
2. Random Forest (moderate, 92-94% accuracy)
3. CNN on spectrograms (complex, 94-96% accuracy)
4. LSTM on pulse sequences (complex, 93-95% accuracy)

Task 4.4: Deployment Strategy

• Train on GigLez server or cloud GPU
• Export model to ONNX for fast inference
• Deploy API endpoint for classification
• Fallback to rule-based if ML unavailable

Expected Accuracy: 90-95% (state-of-the-art)

---

Files Created/Modified

New Files:

• docs/RF_SIGNAL_ANALYSIS_RESEARCH.md - Comprehensive research doc (1,100+ lines)
• scripts/parse_rtl433_devices.py - RTL_433 parser (350 lines)
• data/rtl_433_protocols.json - Device database (4,500 lines, 286 devices)
• static/js/detail-modal.js - Enhanced (with debug logging)
• templates/index.html - Added Leaflet.heat plugin, fixed modal
• static/js/map.js - Added heatmap rendering

Modified Files:

• Heatmap implementation in map.js
• Modal fixes in detail-modal.js and index.html

---

Metrics & Goals

Current State (Before RTL_433):

• Accuracy: ~60-70%
• Known protocols: 20
• Device database size: ~50 entries
• RAW capture handling: Poor (generic matches only)

Phase 1 Complete (Parser Ready):

• RTL_433 protocols parsed: 286
• Database generated: ✅ Yes
• Parser tested: ✅ Yes
• Integration status: 🔄 Next step

Phase 2 Target (After Integration):

• Accuracy: ~75-80% (+15%)
• Known protocols: 286 (14x increase)
• RAW capture handling: Improved with timing
• Timeline: 1-2 weeks

Phase 3 Target (Timing Analysis):

• Accuracy: ~80-85% (+5%)
• RAW captures: 75%+ identification rate
• Timeline: 2-3 weeks

Phase 4+ Target (ML):

• Accuracy: 90-95% (state-of-the-art)
• Timeline: 3-6 months (requires dataset collection)

---

Commands to Run

Re-parse RTL_433 (if needed):

python3 scripts/parse_rtl433_devices.py /tmp/rtl_433/

View database summary:

jq '.total_devices, .devices | group_by(.category) | map({category: .[0].category, count: length})' data/rtl_433_protocols.json

Count by manufacturer:

jq '[.devices[].manufacturer] | group_by(.) | map({manufacturer: .[0], count: length}) | sort_by(-.count) | .[0:10]' data/rtl_433_protocols.json

---

References

• RTL_433 Repository: https://github.com/merbanan/rtl_433
• RTL_433 Documentation: https://github.com/merbanan/rtl_433/tree/master/docs
• Device List: rtl_433/include/rtl_433_devices.h
• Device Sources: rtl_433/src/devices/*.c (255 files)

---

Success Criteria

Phase 1: ✅ COMPLETE

• Parse RTL_433 C source code
• Extract 250+ device protocols
• Generate structured JSON database
• Document all findings
• Commit and push

Phase 2: 🔄 IN PROGRESS

• Create RTL_433 matcher class
• Integrate into existing matcher
• Test with sample captures
• Measure accuracy improvement
• Document results

Phase 3: ⏳ PENDING

• Implement RAW data parser
• Create timing matcher
• Integrate timing analysis
• Test with RAW captures
• Target 80-85% accuracy

---

Status: Phase 1 complete! Ready to proceed with Phase 2 integration.

Estimated Impact: +15-20% accuracy improvement once integrated.

Next Action: Create src/matcher/rtl433_matcher.py and integrate into main matcher.