giglez/docs/WARDRIVER_ONBOARDING.md

# GigLez: Wardriver Onboarding Brief

**TL;DR:** We're building Wigle.net for Sub-GHz IoT devices. Upload RF captures (.sub files) + GPS → auto-identify devices → map IoT infrastructure globally.

---

## What is GigLez?

**Crowdsourced RF IoT Device Mapping Platform**

- **Like Wigle:** Users upload captures → database grows → community maps infrastructure
- **Unlike Wigle:** Instead of WiFi/BT (2.4GHz), we focus on **Sub-GHz IoT** (315/433/868/915 MHz)
- **Devices:** Weather sensors, garage openers, tire pressure monitors, doorbells, security sensors

---

## Why Sub-GHz?

**Massive blind spot in IoT security/research:**
- 433 MHz = most popular IoT frequency (weather, remotes, sensors)
- No centralized database like Wigle (WiFi) or Shodan (internet)
- Devices broadcast constantly, no encryption, easy to capture
- Security implications: replay attacks, device tracking, privacy leaks

**Your wardriving experience transfers perfectly:**
- GPS logging → same workflow
- Signal capture → Flipper Zero instead of WiFi adapter
- Upload interface → familiar Wigle-style submission
- Mapping → identical visualization approach

---

## Current System Architecture

```
User uploads .sub file + GPS
         ↓
  Parse RF Signal
  (frequency, pulses, timing)
         ↓
  Multi-Decoder Pipeline:
    1. RTL_433 (200+ protocols)
    2. Pattern Decoder (timing analysis)
         ↓
  Device Identified
  (LaCrosse TX141, Acurite 5n1, etc.)
         ↓
  Store in PostGIS Database
         ↓
  Display on Interactive Map
```

---

## Device Identification Algorithm (Current)

### Decoder 1: RTL_433 Integration
**What:** Subprocess wrapper around RTL_433 binary (FOSS, 15+ years development)
**Protocols:** 286 devices (weather, automotive, security)
**Method:**
- Converts .sub → RTL_433 pulse format
- Runs protocol matchers (one per device type)
- Returns JSON if decoded

**Performance:**
- Speed: <100ms
- Accuracy: 95% for known protocols
- Limitation: Requires multi-transmission captures (RTL_433 expects repetitions)

**Location:** `src/matcher/rtl433_decoder.py`

### Decoder 2: Pattern-Based Heuristics
**What:** Custom timing pattern analyzer for single-transmission captures
**Method:**
1. **Timing Extraction:** K-means cluster pulse widths → SHORT/LONG identification
2. **Binary Decoding:** Convert pulses to bits (SHORT=0, LONG=1 for PWM)
3. **Statistical Fingerprint:** Calculate mean pulse, duty cycle, pulse-gap ratio, pulse count
4. **Database Matching:** Compare against 25+ protocol signatures
5. **Confidence Scoring:** Weighted by timing accuracy (40%), bit count (30%), pattern match (30%)

**Performance:**
- Speed: <50ms
- Accuracy: 65% for protocol DB, 5% for unknowns
- Works on: Flipper Zero short captures (1 button press)

**Location:** `src/matcher/pattern_decoder.py`

---

## Open-Source Resources Available

### 1. RTL_433 Protocol Database
**Source:** https://github.com/merbanan/rtl_433
- **286 device protocols** with timing signatures
- **JSON export:** `data/rtl_433_protocols.json`
- **Fields:** short_width, long_width, gap_limit, reset_limit, modulation
- **Categories:** Weather (majority), automotive TPMS, security, doorbells

### 2. Flipper Zero .sub File Collections
**Source:** Zero-Sploit/FlipperZero-Subghz-DB (13,717 files)
- Community-contributed signal captures
- Organized by device type
- RAW pulse data + metadata
- **Limitation:** Most labeled by remote function, not device model

### 3. FCC Equipment Authorization Database
**Source:** https://fccid.io/
- **All RF devices sold in US** must be certified
- Contains: Operating frequency, power, device photos, manuals
- **Use case:** Cross-reference identified devices, validate frequency ranges
- **API:** Available for bulk lookups

### 4. GigLez Protocol Database
**Source:** `src/matcher/protocol_database.py`
- **25 hand-curated protocols** extracted from Flipper firmware + RTL_433
- **Detailed timing:** Short/long pulse widths, preamble patterns, sync words
- **Categories:** Weather (7), garage openers (3), doorbells (1), TPMS (2), security (1), remotes (6)

---

## Proposed Algorithm Improvements

### Problem 1: Low Identification Rate for Unknown Devices
**Current:** 5% accuracy on devices not in protocol database
**Impact:** Most user uploads return "Unknown"

### Problem 2: Single-Transmission Weakness
**Current:** RTL_433 needs repetitions, pattern decoder struggles with noise
**Impact:** Flipper captures (1 button press) often fail

### Problem 3: No Learning from User Feedback
**Current:** System static, doesn't improve over time
**Impact:** Missed opportunity to crowd-source knowledge

---

## Improved Heuristic Algorithm (FOSS-Only)

### Enhancement 1: Multi-Pass Timing Analysis

**Current Approach:**
```python
# Single K-means clustering on all pulses
pulses = [520, 1040, 480, 1020, ...]
short, long = kmeans(pulses, k=2)  # Assumes 2 distinct widths
```

**Improved Approach:**
```python
# Hierarchical clustering + outlier removal
def extract_timing_robust(pulses):
    # Step 1: Remove outliers (noise, glitches)
    pulses_clean = remove_outliers(pulses, method='IQR')

    # Step 2: Separate HIGH vs LOW pulses
    high_pulses = [p for p in pulses if p > 0]
    low_pulses = [abs(p) for p in pulses if p < 0]

    # Step 3: Multi-level clustering
    # Try k=2,3,4 (some protocols have SHORT/MID/LONG)
    for k in [2, 3, 4]:
        clusters = kmeans(high_pulses, k=k)
        if is_valid_clustering(clusters):  # Check separation
            return clusters

    # Step 4: Frequency histogram method (fallback)
    return histogram_peaks(high_pulses)
```

**Benefit:** Handles multi-level modulation (e.g., tri-bit encoding)

### Enhancement 2: Frequency-Based Protocol Filtering

**Current:** Search all 286 RTL_433 protocols
**Improved:** Pre-filter by frequency band

```python
FREQUENCY_PROTOCOL_MAP = {
    433920000: {  # 433.92 MHz ISM band
        'weather': [12, 19, 20, 32, 40, 55, 73, 113],  # RTL_433 protocol IDs
        'garage': [1, 8, 9],
        'security': [25, 26],
    },
    315000000: {  # 315 MHz (North America)
        'automotive': [10, 11, 40],  # TPMS
        'garage': [22, 23],
    },
    868000000: {  # 868 MHz SRD (Europe)
        'weather': [78, 88, 113],
        'home_automation': [95, 102],
    }
}

def filter_protocols_by_frequency(freq, tolerance=100_000):
    """Return likely protocol IDs based on frequency"""
    freq_band = round_to_nearest_band(freq)
    return FREQUENCY_PROTOCOL_MAP.get(freq_band, [])
```

**Benefit:** 10x speedup (test 20 protocols instead of 200)

### Enhancement 3: Preamble/Sync Pattern Detection

**Current:** Only checks if preamble exists in decoded bits
**Improved:** Dedicated preamble detector before decoding

```python
def detect_preamble(pulses):
    """
    Preambles are repeating patterns at start of transmission
    Examples:
      - Oregon Scientific: 16x "10" = 32 alternating pulses
      - Princeton: 4x "1111" = long HIGH burst
    """
    # Check first 50 pulses for repetition
    first_50 = pulses[:50]

    # Method 1: Autocorrelation for periodic patterns
    period = find_autocorrelation_peak(first_50)
    if period:
        pattern = first_50[:period]
        repetitions = count_repetitions(first_50, pattern)
        if repetitions >= 4:
            return {
                'type': 'periodic',
                'pattern_length': period,
                'repetitions': repetitions
            }

    # Method 2: Long burst detection (e.g., "1111...")
    if first_50[0] > mean(first_50) * 2:  # First pulse much longer
        return {'type': 'long_burst', 'duration': first_50[0]}

    return None
```

**Benefit:** Narrow down protocols before full decode (faster + more accurate)

### Enhancement 4: Protocol Signature Expansion

**Current:** 25 protocols in `protocol_database.py`
**Target:** Expand to 100+ using RTL_433 JSON

**Automated Extraction Script:**
```python
def import_rtl433_protocols():
    """
    Parse RTL_433 source code to extract timing signatures

    RTL_433 C code format:
        .short_width = 500,
        .long_width = 1000,
        .gap_limit = 2000,
        .reset_limit = 5000,
    """
    rtl433_repo = "~/rtl_433/src/devices/"
    protocols = []

    for c_file in glob(f"{rtl433_repo}/*.c"):
        # Regex extraction from C structs
        signature = extract_timing_from_c(c_file)
        if signature:
            protocols.append(ProtocolSignature(
                name=signature['name'],
                short_pulse_us=signature['short_width'],
                long_pulse_us=signature['long_width'],
                # ... more fields
            ))

    return protocols
```

**Benefit:** 4x larger protocol database (25 → 100+), no manual curation

### Enhancement 5: Device Disambiguation via Metadata

**Problem:** Multiple devices have identical timing (e.g., Princeton = generic chipset)
**Solution:** Use secondary characteristics

```python
def disambiguate_matches(matches, metadata):
    """
    Rank matches using:
      1. Frequency exact match (higher weight)
      2. Bit count exact match
      3. Preamble pattern match
      4. Geographic prior (common devices in region)
    """
    scored = []
    for match in matches:
        score = match.confidence

        # Bonus: Exact frequency match
        if abs(match.frequency - metadata.frequency) < 10_000:
            score *= 1.2

        # Bonus: Bit count perfect match
        bit_count = len(metadata.decoded_bits)
        if match.min_bits <= bit_count <= match.max_bits:
            if bit_count == match.typical_bits:
                score *= 1.15

        # Bonus: Preamble detected and matches
        if metadata.preamble and match.preamble_pattern:
            if metadata.preamble.startswith(match.preamble_pattern):
                score *= 1.3

        # Bonus: Common in user's region (from GPS)
        if metadata.gps:
            regional_devices = get_common_devices_nearby(metadata.gps)
            if match.name in regional_devices:
                score *= 1.1

        scored.append((match, score))

    return sorted(scored, key=lambda x: x[1], reverse=True)
```

**Benefit:** Princeton @ 433MHz + 24-bit → could be garage opener OR remote → GPS (residential area) → likely garage opener

---

## Data Sources for Protocol Expansion

### Source 1: RTL_433 Device C Files
**Path:** https://github.com/merbanan/rtl_433/tree/master/src/devices
**Count:** 286 .c files
**Extractable Data:**
- Timing parameters (short/long/gap/reset widths)
- Modulation type (OOK/FSK)
- Bit lengths
- Manufacturer/model names

**Extraction Method:** Regex parsing of C structs

### Source 2: Flipper Zero Firmware
**Path:** https://github.com/flipperdevices/flipperzero-firmware/tree/dev/lib/subghz/protocols
**Count:** ~40 protocol decoders
**Extractable Data:**
- Timing tolerances
- Encoding schemes (PWM, Manchester, etc.)
- Preamble patterns
- Sample data payloads

**Extraction Method:** Parse C protocol definitions

### Source 3: Universal Radio Hacker (URH)
**Path:** https://github.com/jopohl/urh
**Tool:** GUI for reverse-engineering RF protocols
**Output:** XML protocol definitions
**Use Case:** Community could contribute URH-analyzed protocols

### Source 4: GigLez User Submissions
**Method:** Crowd-source unknown signals
**Workflow:**
1. User uploads "Unknown" capture
2. Admin/community analyzes with URH or manual tools
3. Creates protocol signature
4. Adds to database → future captures auto-matched

**Gamification:** Leaderboard for protocol contributors (like Wigle)

---

## Implementation Priority

### Phase 1: Protocol Database Expansion (Week 1)
- [ ] Parse RTL_433 JSON → extract 100+ additional signatures
- [ ] Import to `protocol_database.py`
- [ ] Test: Does this improve accuracy on test dataset?

### Phase 2: Improved Timing Analysis (Week 2)
- [ ] Implement robust clustering with outlier removal
- [ ] Add multi-level clustering (k=2,3,4)
- [ ] Preamble detection algorithm
- [ ] Benchmark: Accuracy on single-transmission captures

### Phase 3: Frequency-Based Filtering (Week 3)
- [ ] Build frequency → protocol ID mapping
- [ ] Integrate with RTL_433 decoder (pass -R flags)
- [ ] Benchmark: Speed improvement

### Phase 4: Disambiguation Logic (Week 4)
- [ ] Implement metadata-based scoring
- [ ] Add geographic priors (query PostGIS for nearby device types)
- [ ] Test: Reduction in ambiguous results

---

## Expected Improvements

| Metric | Current | Target | Method |
|--------|---------|--------|--------|
| **Protocol DB Size** | 25 | 100+ | Auto-extract RTL_433 |
| **Unknown Device Accuracy** | 5% | 40% | Expanded DB + robust timing |
| **Single-Tx Success Rate** | 20% | 65% | Outlier removal + preamble detection |
| **Disambiguation Accuracy** | 60% | 85% | Metadata scoring |
| **Avg Processing Time** | 150ms | 80ms | Frequency filtering |

---

## How You Can Contribute

### As Wardriver with Data:
1. **Upload Captures:** If you have Flipper Zero, start wardrive-style captures
   - Walk/drive with Flipper in "Read RAW" mode
   - Save .sub files with GPS timestamps
   - Bulk upload via API or web interface

2. **Verify Identifications:** Review auto-matched devices
   - Confirm: "Yes, that's a LaCrosse sensor"
   - Correct: "No, it's an Acurite 5n1"
   - Feedback trains future improvements

3. **Map Coverage:** Apply Wigle strategy
   - Focus on under-mapped areas
   - Multiple passes for verification
   - Track unique devices vs observations

### As RF/Protocol Expert:
1. **Protocol Analysis:** Help identify unknowns
   - Use Universal Radio Hacker (URH)
   - Document timing patterns
   - Submit to protocol database

2. **Algorithm Tuning:** Test decoder variants
   - Different clustering methods
   - Thresholds for confidence scoring
   - Edge cases (noise, interference)

3. **Dataset Creation:** Build labeled test set
   - Purchase 10-20 common devices
   - Capture ground-truth signals
   - Use for accuracy benchmarking

---

## Quick Start

### View Current System:
```bash
cd /home/dell/coding/giglez

# See protocol database
python src/matcher/protocol_database.py

# Test pattern decoder
python src/matcher/pattern_decoder.py

# Check RTL_433 integration
python src/matcher/rtl433_decoder.py
```

### Test on Sample Data:
```bash
# We have RTL_433 test captures
ls data/rf_test_datasets/rtl_433_tests/tests/

# Example: Decode a weather sensor
python scripts/test_rtl433_with_known_devices.py
```

### Database Schema:
```sql
-- Key tables
captures (id, lat, lon, timestamp, frequency, file_path, pulse_count)
devices (id, name, manufacturer, category, frequency)
identifications (capture_id, device_id, confidence, method)
```

---

## Questions?

**Codebase:** `/home/dell/coding/giglez/`
**Docs:**
- `docs/ML_DESIGN.md` - Full ML research (ignore if focusing on heuristics)
- `CLAUDE.md` - Project overview
- `docs/IMPLEMENTATION_SUMMARY.md` - Current progress

**Similar Projects:**
- Wigle.net (WiFi wardriving) - our inspiration
- RTL_433 (device decoder) - our decoder backend
- Flipper Zero (capture tool) - our data source

**Next Steps:**
1. Review protocol database expansion script (Phase 1)
2. Discuss disambiguation heuristics (Phase 4)
3. Define accuracy benchmarks for success criteria

---

**Document Version:** 1.0
**Last Updated:** 2026-02-14
**Status:** Ready for collaborator review