giglez/docs/WARDRIVER_ONBOARDING.md

GigLez: Wardriver Onboarding Brief

TL;DR: We're building Wigle.net for Sub-GHz IoT devices. Upload RF captures (.sub files) + GPS → auto-identify devices → map IoT infrastructure globally.

---

What is GigLez?

Crowdsourced RF IoT Device Mapping Platform

• Like Wigle: Users upload captures → database grows → community maps infrastructure
• Unlike Wigle: Instead of WiFi/BT (2.4GHz), we focus on Sub-GHz IoT (315/433/868/915 MHz)
• Devices: Weather sensors, garage openers, tire pressure monitors, doorbells, security sensors

---

Why Sub-GHz?

Massive blind spot in IoT security/research:

• 433 MHz = most popular IoT frequency (weather, remotes, sensors)
• No centralized database like Wigle (WiFi) or Shodan (internet)
• Devices broadcast constantly, no encryption, easy to capture
• Security implications: replay attacks, device tracking, privacy leaks

Your wardriving experience transfers perfectly:

• GPS logging → same workflow
• Signal capture → Flipper Zero instead of WiFi adapter
• Upload interface → familiar Wigle-style submission
• Mapping → identical visualization approach

---

Current System Architecture

User uploads .sub file + GPS
         ↓
  Parse RF Signal
  (frequency, pulses, timing)
         ↓
  Multi-Decoder Pipeline:
    1. RTL_433 (200+ protocols)
    2. Pattern Decoder (timing analysis)
         ↓
  Device Identified
  (LaCrosse TX141, Acurite 5n1, etc.)
         ↓
  Store in PostGIS Database
         ↓
  Display on Interactive Map

---

Device Identification Algorithm (Current)

Decoder 1: RTL_433 Integration

What: Subprocess wrapper around RTL_433 binary (FOSS, 15+ years development) Protocols: 286 devices (weather, automotive, security) Method:

• Converts .sub → RTL_433 pulse format
• Runs protocol matchers (one per device type)
• Returns JSON if decoded

Performance:

• Speed: <100ms
• Accuracy: 95% for known protocols
• Limitation: Requires multi-transmission captures (RTL_433 expects repetitions)

Location: src/matcher/rtl433_decoder.py

Decoder 2: Pattern-Based Heuristics

What: Custom timing pattern analyzer for single-transmission captures Method:

1. Timing Extraction: K-means cluster pulse widths → SHORT/LONG identification
2. Binary Decoding: Convert pulses to bits (SHORT=0, LONG=1 for PWM)
3. Statistical Fingerprint: Calculate mean pulse, duty cycle, pulse-gap ratio, pulse count
4. Database Matching: Compare against 25+ protocol signatures
5. Confidence Scoring: Weighted by timing accuracy (40%), bit count (30%), pattern match (30%)

Performance:

• Speed: <50ms
• Accuracy: 65% for protocol DB, 5% for unknowns
• Works on: Flipper Zero short captures (1 button press)

Location: src/matcher/pattern_decoder.py

---

Open-Source Resources Available

1. RTL_433 Protocol Database

Source: https://github.com/merbanan/rtl_433

• 286 device protocols with timing signatures
• JSON export: data/rtl_433_protocols.json
• Fields: short_width, long_width, gap_limit, reset_limit, modulation
• Categories: Weather (majority), automotive TPMS, security, doorbells

2. Flipper Zero .sub File Collections

Source: Zero-Sploit/FlipperZero-Subghz-DB (13,717 files)

• Community-contributed signal captures
• Organized by device type
• RAW pulse data + metadata
• Limitation: Most labeled by remote function, not device model

3. FCC Equipment Authorization Database

Source: https://fccid.io/

• All RF devices sold in US must be certified
• Contains: Operating frequency, power, device photos, manuals
• Use case: Cross-reference identified devices, validate frequency ranges
• API: Available for bulk lookups

4. GigLez Protocol Database

Source: src/matcher/protocol_database.py

• 25 hand-curated protocols extracted from Flipper firmware + RTL_433
• Detailed timing: Short/long pulse widths, preamble patterns, sync words
• Categories: Weather (7), garage openers (3), doorbells (1), TPMS (2), security (1), remotes (6)

---

Proposed Algorithm Improvements

Problem 1: Low Identification Rate for Unknown Devices

Current: 5% accuracy on devices not in protocol database Impact: Most user uploads return "Unknown"

Problem 2: Single-Transmission Weakness

Current: RTL_433 needs repetitions, pattern decoder struggles with noise Impact: Flipper captures (1 button press) often fail

Problem 3: No Learning from User Feedback

Current: System static, doesn't improve over time Impact: Missed opportunity to crowd-source knowledge

---

Improved Heuristic Algorithm (FOSS-Only)

Enhancement 1: Multi-Pass Timing Analysis

Current Approach:

# Single K-means clustering on all pulses
pulses = [520, 1040, 480, 1020, ...]
short, long = kmeans(pulses, k=2)  # Assumes 2 distinct widths

Improved Approach:

# Hierarchical clustering + outlier removal
def extract_timing_robust(pulses):
    # Step 1: Remove outliers (noise, glitches)
    pulses_clean = remove_outliers(pulses, method='IQR')

    # Step 2: Separate HIGH vs LOW pulses
    high_pulses = [p for p in pulses if p > 0]
    low_pulses = [abs(p) for p in pulses if p < 0]

    # Step 3: Multi-level clustering
    # Try k=2,3,4 (some protocols have SHORT/MID/LONG)
    for k in [2, 3, 4]:
        clusters = kmeans(high_pulses, k=k)
        if is_valid_clustering(clusters):  # Check separation
            return clusters

    # Step 4: Frequency histogram method (fallback)
    return histogram_peaks(high_pulses)

Benefit: Handles multi-level modulation (e.g., tri-bit encoding)

Enhancement 2: Frequency-Based Protocol Filtering

Current: Search all 286 RTL_433 protocols Improved: Pre-filter by frequency band

FREQUENCY_PROTOCOL_MAP = {
    433920000: {  # 433.92 MHz ISM band
        'weather': [12, 19, 20, 32, 40, 55, 73, 113],  # RTL_433 protocol IDs
        'garage': [1, 8, 9],
        'security': [25, 26],
    },
    315000000: {  # 315 MHz (North America)
        'automotive': [10, 11, 40],  # TPMS
        'garage': [22, 23],
    },
    868000000: {  # 868 MHz SRD (Europe)
        'weather': [78, 88, 113],
        'home_automation': [95, 102],
    }
}

def filter_protocols_by_frequency(freq, tolerance=100_000):
    """Return likely protocol IDs based on frequency"""
    freq_band = round_to_nearest_band(freq)
    return FREQUENCY_PROTOCOL_MAP.get(freq_band, [])

Benefit: 10x speedup (test 20 protocols instead of 200)

Enhancement 3: Preamble/Sync Pattern Detection

Current: Only checks if preamble exists in decoded bits Improved: Dedicated preamble detector before decoding

def detect_preamble(pulses):
    """
    Preambles are repeating patterns at start of transmission
    Examples:
      - Oregon Scientific: 16x "10" = 32 alternating pulses
      - Princeton: 4x "1111" = long HIGH burst
    """
    # Check first 50 pulses for repetition
    first_50 = pulses[:50]

    # Method 1: Autocorrelation for periodic patterns
    period = find_autocorrelation_peak(first_50)
    if period:
        pattern = first_50[:period]
        repetitions = count_repetitions(first_50, pattern)
        if repetitions >= 4:
            return {
                'type': 'periodic',
                'pattern_length': period,
                'repetitions': repetitions
            }

    # Method 2: Long burst detection (e.g., "1111...")
    if first_50[0] > mean(first_50) * 2:  # First pulse much longer
        return {'type': 'long_burst', 'duration': first_50[0]}

    return None

Benefit: Narrow down protocols before full decode (faster + more accurate)

Enhancement 4: Protocol Signature Expansion

Current: 25 protocols in protocol_database.py Target: Expand to 100+ using RTL_433 JSON

Automated Extraction Script:

def import_rtl433_protocols():
    """
    Parse RTL_433 source code to extract timing signatures

    RTL_433 C code format:
        .short_width = 500,
        .long_width = 1000,
        .gap_limit = 2000,
        .reset_limit = 5000,
    """
    rtl433_repo = "~/rtl_433/src/devices/"
    protocols = []

    for c_file in glob(f"{rtl433_repo}/*.c"):
        # Regex extraction from C structs
        signature = extract_timing_from_c(c_file)
        if signature:
            protocols.append(ProtocolSignature(
                name=signature['name'],
                short_pulse_us=signature['short_width'],
                long_pulse_us=signature['long_width'],
                # ... more fields
            ))

    return protocols

Benefit: 4x larger protocol database (25 → 100+), no manual curation

Enhancement 5: Device Disambiguation via Metadata

Problem: Multiple devices have identical timing (e.g., Princeton = generic chipset) Solution: Use secondary characteristics

def disambiguate_matches(matches, metadata):
    """
    Rank matches using:
      1. Frequency exact match (higher weight)
      2. Bit count exact match
      3. Preamble pattern match
      4. Geographic prior (common devices in region)
    """
    scored = []
    for match in matches:
        score = match.confidence

        # Bonus: Exact frequency match
        if abs(match.frequency - metadata.frequency) < 10_000:
            score *= 1.2

        # Bonus: Bit count perfect match
        bit_count = len(metadata.decoded_bits)
        if match.min_bits <= bit_count <= match.max_bits:
            if bit_count == match.typical_bits:
                score *= 1.15

        # Bonus: Preamble detected and matches
        if metadata.preamble and match.preamble_pattern:
            if metadata.preamble.startswith(match.preamble_pattern):
                score *= 1.3

        # Bonus: Common in user's region (from GPS)
        if metadata.gps:
            regional_devices = get_common_devices_nearby(metadata.gps)
            if match.name in regional_devices:
                score *= 1.1

        scored.append((match, score))

    return sorted(scored, key=lambda x: x[1], reverse=True)

Benefit: Princeton @ 433MHz + 24-bit → could be garage opener OR remote → GPS (residential area) → likely garage opener

---

Data Sources for Protocol Expansion

Source 1: RTL_433 Device C Files

Path: https://github.com/merbanan/rtl_433/tree/master/src/devices Count: 286 .c files Extractable Data:

• Timing parameters (short/long/gap/reset widths)
• Modulation type (OOK/FSK)
• Bit lengths
• Manufacturer/model names

Extraction Method: Regex parsing of C structs

Source 2: Flipper Zero Firmware

Path: https://github.com/flipperdevices/flipperzero-firmware/tree/dev/lib/subghz/protocols Count: ~40 protocol decoders Extractable Data:

• Timing tolerances
• Encoding schemes (PWM, Manchester, etc.)
• Preamble patterns
• Sample data payloads

Extraction Method: Parse C protocol definitions

Source 3: Universal Radio Hacker (URH)

Path: https://github.com/jopohl/urh Tool: GUI for reverse-engineering RF protocols Output: XML protocol definitions Use Case: Community could contribute URH-analyzed protocols

Source 4: GigLez User Submissions

Method: Crowd-source unknown signals Workflow:

1. User uploads "Unknown" capture
2. Admin/community analyzes with URH or manual tools
3. Creates protocol signature
4. Adds to database → future captures auto-matched

Gamification: Leaderboard for protocol contributors (like Wigle)

---

Implementation Priority

Phase 1: Protocol Database Expansion (Week 1)

• Parse RTL_433 JSON → extract 100+ additional signatures
• Import to protocol_database.py
• Test: Does this improve accuracy on test dataset?

Phase 2: Improved Timing Analysis (Week 2)

• Implement robust clustering with outlier removal
• Add multi-level clustering (k=2,3,4)
• Preamble detection algorithm
• Benchmark: Accuracy on single-transmission captures

Phase 3: Frequency-Based Filtering (Week 3)

• Build frequency → protocol ID mapping
• Integrate with RTL_433 decoder (pass -R flags)
• Benchmark: Speed improvement

Phase 4: Disambiguation Logic (Week 4)

• Implement metadata-based scoring
• Add geographic priors (query PostGIS for nearby device types)
• Test: Reduction in ambiguous results

---

Expected Improvements

Metric	Current	Target	Method
Protocol DB Size	25	100+	Auto-extract RTL_433
Unknown Device Accuracy	5%	40%	Expanded DB + robust timing
Single-Tx Success Rate	20%	65%	Outlier removal + preamble detection
Disambiguation Accuracy	60%	85%	Metadata scoring
Avg Processing Time	150ms	80ms	Frequency filtering

---

How You Can Contribute

As Wardriver with Data:

1. Upload Captures: If you have Flipper Zero, start wardrive-style captures

   • Walk/drive with Flipper in "Read RAW" mode
   • Save .sub files with GPS timestamps
   • Bulk upload via API or web interface

2. Verify Identifications: Review auto-matched devices

   • Confirm: "Yes, that's a LaCrosse sensor"
   • Correct: "No, it's an Acurite 5n1"
   • Feedback trains future improvements

3. Map Coverage: Apply Wigle strategy

   • Focus on under-mapped areas
   • Multiple passes for verification
   • Track unique devices vs observations

As RF/Protocol Expert:

1. Protocol Analysis: Help identify unknowns

   • Use Universal Radio Hacker (URH)
   • Document timing patterns
   • Submit to protocol database

2. Algorithm Tuning: Test decoder variants

   • Different clustering methods
   • Thresholds for confidence scoring
   • Edge cases (noise, interference)

3. Dataset Creation: Build labeled test set

   • Purchase 10-20 common devices
   • Capture ground-truth signals
   • Use for accuracy benchmarking

---

Quick Start

View Current System:

cd /home/dell/coding/giglez

# See protocol database
python src/matcher/protocol_database.py

# Test pattern decoder
python src/matcher/pattern_decoder.py

# Check RTL_433 integration
python src/matcher/rtl433_decoder.py

Test on Sample Data:

# We have RTL_433 test captures
ls data/rf_test_datasets/rtl_433_tests/tests/

# Example: Decode a weather sensor
python scripts/test_rtl433_with_known_devices.py

Database Schema:

-- Key tables
captures (id, lat, lon, timestamp, frequency, file_path, pulse_count)
devices (id, name, manufacturer, category, frequency)
identifications (capture_id, device_id, confidence, method)

---

Questions?

Codebase: /home/dell/coding/giglez/ Docs:

• docs/ML_DESIGN.md - Full ML research (ignore if focusing on heuristics)
• CLAUDE.md - Project overview
• docs/IMPLEMENTATION_SUMMARY.md - Current progress

Similar Projects:

• Wigle.net (WiFi wardriving) - our inspiration
• RTL_433 (device decoder) - our decoder backend
• Flipper Zero (capture tool) - our data source

Next Steps:

1. Review protocol database expansion script (Phase 1)
2. Discuss disambiguation heuristics (Phase 4)
3. Define accuracy benchmarks for success criteria

---

Document Version: 1.0 Last Updated: 2026-02-14 Status: Ready for collaborator review