giglez/docs/WARDRIVER_ONBOARDING.md

# GigLez - RF Wardriver Onboarding & Data Format Integration

**Welcome, RF Wardrivers!** This document explains what we've built, current system capabilities, and **critically, what data formats we need to support for your existing wardriving workflows**.

---

## What We've Built: GigLez Overview

**GigLez** is a Wigle-style crowdsourced platform for mapping Sub-GHz IoT RF devices (300-928 MHz). Think "Wigle.net but for garage doors, weather stations, tire pressure monitors, and doorbells instead of WiFi."

### Core Concept
- **Upload** .sub files (Flipper Zero format) with GPS coordinates
- **Automatic Device Identification** using RF signature matching (299 protocol database)
- **Interactive Map** showing IoT device distribution by location
- **Community Verification** for device IDs and crowdsourced improvements

---

## What's Working Right Now

### ✅ Device Identification System (6-Component Scoring)

We've built a multi-factor RF signature matching engine with **33% accuracy on synthetic signals** (baseline established):

| Component | Weight | Purpose |
|-----------|--------|---------|
| **Timing Analysis** | 40% | K-means clustering to extract SHORT/LONG pulse widths |
| **Preamble Detection** | 25% | Identifies sync patterns (alternating, long_burst, sync_word) |
| **Timing Ratio** | 20% | Discriminates protocols by LONG/SHORT ratio (2:1 vs 3:1) |
| **Frequency Fingerprinting** | 10% | ISM band classification (315/433/868/915 MHz, ±100kHz tolerance) |
| **Bit Count Matching** | 5% | Validates bit length against protocol expectations |

**Processing Speed**: 156ms average per signal (0.4ms parse + 156ms matching)

**Supported Modulations**: OOK, FSK (from .sub metadata)

### ✅ Protocol Database (299 Protocols)

Imported from RTL_433 project:
- **Weather Sensors**: LaCrosse, Oregon Scientific, Acurite, Nexus (70+ protocols)
- **Garage Doors**: Chamberlain, LiftMaster, Linear (15+ protocols)
- **TPMS**: Schrader, Toyota, Honda, Ford (30+ protocols)
- **Security**: Honeywell, SimpliSafe, DSC (20+ protocols)
- **Doorbells & Remotes**: Princeton, PT2262, EV1527 (40+ protocols)
- **Lighting/Climate**: LED controllers, ceiling fans, thermostats (20+ protocols)

**Frequency Coverage**:
- 315 MHz: 15% (garage doors, car keys)
- 433 MHz: 70% (weather, doorbells, generic remotes)
- 868 MHz: 10% (EU devices)
- 915 MHz: 5% (US ISM band)

### ✅ GPS Validation & Privacy Features

**Implemented** (`src/gps/validator.py`):
- Latitude/longitude validation (-90 to 90, -180 to 180)
- Coordinate precision rounding (configurable 4-8 decimal places)
- Altitude validation (optional, -500m to 10000m)
- Haversine distance calculation for duplicate detection
- **Privacy**: GPS anonymization via coordinate rounding

**Example**:
```python
from src.gps.validator import validate_coordinates, anonymize_location

# Validate raw GPS
coords = validate_coordinates(lat=40.712776, lon=-74.005974, altitude=10.5)

# Anonymize (round to ~11m precision)
anon = anonymize_location(lat=40.712776, lon=-74.005974, precision=4)
# Returns: (40.7128, -74.0060)
```

### ✅ File Format Parsing (.sub files)

**Currently Supported**: Flipper Zero .sub format (both RAW and decoded variants)

**Parser** (`src/parser/sub_parser.py`):
- Extracts frequency, preset (modulation), protocol name
- Parses RAW_Data into pulse sequences
- Handles both OOK and FSK modulation metadata
- Validates file structure

**Example .sub file**:
```
Filetype: Flipper SubGhz RAW File
Version: 1
Frequency: 433920000
Preset: FuriHalSubGhzPresetOok650Async
Protocol: RAW
RAW_Data: 2980 -240 520 -980 520 -980 1000 -500 ...
```

### ⚠️ Test Coverage

- **Unit Tests**: 56 tests passing (timing, preamble, frequency, GPS, database)
- **Synthetic Benchmark**: 33% top-1 accuracy on 12 test signals
- **Real-World Benchmark**: **0% accuracy on 11 real Flipper captures** ❌

---

## Critical Gap: Real-World Performance Issue

### The Problem: Synthetic vs Real

We optimized for synthetic test signals, but **real Flipper Zero community captures fail completely**.

**Root Causes** (analyzed in `docs/REAL_CAPTURE_ANALYSIS.md`):

1. **Decoded vs RAW Format** (45% of failures)
   - Many real .sub files are **already decoded** (`Protocol: Holtek_HT12X`)
   - Our system only processes `RAW_Data` fields
   - **Fix Needed**: Parse decoded .sub files by protocol name + timing element (TE)

2. **Missing Protocols** (27% of failures)
   - Database lacks: Acurite 02077M, GE Doorbell 19297, Byron DB421E, Holtek HT12X
   - **Fix Needed**: Import from real captures or community contributions

3. **Signal Complexity** (27% of failures)
   - Real captures have multi-packet transmissions (131 pulses vs 40 bits)
   - Variable signal quality, interference beyond 15% jitter tolerance
   - **Fix Needed**: Packet segmentation, higher noise tolerance (15% → 25%)

4. **Protocol Family Competition** (9% of failures)
   - Nexus ranks #34, beaten by similar Oregon Scientific protocols
   - **Fix Needed**: Protocol family grouping and boosting

---

## What We Need From You: Data Format Questions

To make GigLez seamless for **existing RF wardriving workflows**, we need to understand your data collection methods and formats.

### 1. Hardware & Capture Tools

**What RF capture hardware do you use?**
- [ ] Flipper Zero
- [ ] RTL-SDR dongles
- [ ] HackRF One
- [ ] LimeSDR
- [ ] YARD Stick One
- [ ] LilyGo T-Watch / LoRa devices
- [ ] Other: _______________

**What software do you use to capture signals?**
- [ ] Flipper Zero firmware (generates .sub files)
- [ ] rtl_433 (outputs JSON/CSV)
- [ ] Universal Radio Hacker (URH) - saves .complex/.cfile
- [ ] GNU Radio (custom flowgraphs)
- [ ] inspectrum (visual analysis)
- [ ] SDR# (saves .wav / IQ recordings)
- [ ] gqrx (IQ recordings)
- [ ] Other: _______________

### 2. File Format Preferences

**What file formats do your captures produce?**
- [ ] .sub (Flipper Zero)
- [ ] .fff (Flipper Zero older format)
- [ ] JSON (rtl_433 output)
- [ ] CSV (rtl_433 CSV mode)
- [ ] .complex / .cfile (URH raw IQ data)
- [ ] .wav (SDR# IQ recordings)
- [ ] .sigmf (SigMF metadata format)
- [ ] .cu8 / .cs8 (raw IQ samples)
- [ ] Other: _______________

**Would you prefer to upload:**
- [ ] Individual files with GPS metadata embedded
- [ ] Batch uploads (ZIP with manifest.json)
- [ ] Real-time streaming (live wardriving session)
- [ ] CSV/JSON logs referencing stored signal files

### 3. GPS Data Association

**How do you currently associate GPS coordinates with captures?**
- [ ] Manual entry (lat/lon typed per file)
- [ ] GPX track log synced by timestamp
- [ ] NMEA GPS logs (separate file)
- [ ] Embedded in capture file metadata
- [ ] Kismet-style CSV (BSSID, lat, lon, timestamp)
- [ ] Custom database/spreadsheet
- [ ] Other: _______________

**GPS Precision Requirements:**
- What precision do you typically capture? (e.g., 6 decimal places = ±0.11m)
- Do you need altitude data stored?
- Do you want GPS anonymization options (round to city-level precision)?

### 4. Metadata & Session Context

**What additional metadata do you typically log?**
- [ ] Device photos (visual identification)
- [ ] Signal strength (RSSI)
- [ ] Capture duration
- [ ] Antenna type/gain
- [ ] Weather conditions
- [ ] Session notes/comments
- [ ] Device orientation (N/S/E/W)
- [ ] Other: _______________

### 5. Batch Upload Workflows

**Typical wardriving session size:**
- How many captures per session? (e.g., 50, 500, 5000+)
- Do you prefer uploading during the drive or post-processing later?
- Would you use a CLI tool for bulk uploads or web interface?

**Preferred batch format:**
```
Option A: ZIP with manifest.json
captures.zip
├── capture_001.sub
├── capture_002.sub
├── ...
└── manifest.json  # GPS + timestamps for all files

Option B: CSV index + separate signal files
session_2025-01-15.csv (columns: filename, lat, lon, timestamp, notes)
signals/
├── capture_001.sub
├── capture_002.sub
└── ...

Option C: Real-time API streaming (JSON payloads)
```

Which format would integrate best with your existing tools?

### 6. Device Identification Preferences

**When you capture unknown signals, what info helps you ID them?**
- [ ] Frequency alone (e.g., "433.92 MHz probably garage door")
- [ ] Timing patterns (pulse widths)
- [ ] Bit patterns (binary data)
- [ ] Physical device photo
- [ ] FCC ID database lookup
- [ ] Community voting/verification
- [ ] Other: _______________

**Confidence threshold for auto-tagging:**
- Would you trust automatic IDs at 80% confidence?
- Would you manually verify all IDs regardless of confidence?
- Prefer "suggested IDs" that you confirm before publishing?

### 7. Privacy & Data Sharing

**Data sharing preferences:**
- [ ] Make all captures public (Wigle-style open database)
- [ ] Private by default (opt-in sharing)
- [ ] Share anonymized location data only (round to city)
- [ ] Share device types but not GPS coordinates
- [ ] Custom privacy controls per capture

**Removal requests:**
- Should users be able to request removal of specific captures (like Wigle's BSSID opt-out)?

---

## Integration Roadmap

Based on your feedback, we'll prioritize:

### Phase 1: Core Format Support
- [ ] rtl_433 JSON import (most common for non-Flipper wardrivers)
- [ ] Decoded .sub file support (fix 45% of current failures)
- [ ] GPX track log parsing

### Phase 2: Batch Upload System
- [ ] ZIP + manifest.json uploader
- [ ] CLI tool for bulk submissions
- [ ] API endpoints for real-time streaming

### Phase 3: Additional Hardware Support
- [ ] URH .complex/.cfile import (convert to timing patterns)
- [ ] SDR# .wav IQ file processing
- [ ] SigMF metadata parsing

### Phase 4: Community Features
- [ ] Manual device ID submission
- [ ] Photo upload for visual verification
- [ ] Voting system for ID accuracy
- [ ] Protocol signature contributions

---

## How to Contribute Your Data Now

### Option 1: Share Sample Files
Send us 5-10 representative captures from your wardriving sessions:
- Include GPS coordinates (lat/lon/timestamp)
- Mix of known and unknown devices
- Note your capture tool and workflow

**Where to send**: [Create GitHub issue](https://github.com/yourusername/giglez/issues) or email

### Option 2: Test the Current System
Upload Flipper Zero .sub files via our API (coming soon):
```bash
curl -X POST https://giglez.io/api/submit \
  -F "file=@capture.sub" \
  -F "latitude=40.7128" \
  -F "longitude=-74.0060" \
  -F "timestamp=2025-01-15T12:00:00Z"
```

### Option 3: Provide rtl_433 JSON
If you use rtl_433, share sample JSON output:
```bash
rtl_433 -F json > captures.json
# Include GPS coordinates for each capture (manual or GPX sync)
```

---

## Questions?

We want to build this **for the wardriving community**, so your input directly shapes development priorities.

**Contact**:
- GitHub Issues: [giglez/issues](https://github.com/yourusername/giglez/issues)
- Email: team@giglez.io (placeholder)
- Discord: [GigLez Community](https://discord.gg/giglez) (placeholder)

---

## Response Template (Copy/Paste & Fill Out)

```
### My Wardriving Setup

**Hardware**: [e.g., HackRF One + GPS dongle]
**Software**: [e.g., rtl_433 + GPX logger]
**File Formats**: [e.g., JSON output from rtl_433]
**GPS Method**: [e.g., GPX track synced by timestamp]
**Session Size**: [e.g., 200-500 captures per drive]
**Upload Preference**: [e.g., Batch upload via CLI after session]

### Data Format Details

[Paste sample file snippet or describe structure]

### Integration Requests

[What would make GigLez fit seamlessly into your workflow?]

### Privacy Preferences

[Public/private/anonymized?]
```

---

**Thank you for helping build the Sub-GHz IoT mapping platform!** 🚗📡

*Last Updated: 2026-02-16*