docs: add RF wardriver onboarding & data format survey
Created comprehensive onboarding document for experienced RF wardrivers to: - Understand GigLez platform capabilities (6-component scoring, 299 protocols, GPS validation) - Review current performance metrics (33% synthetic, 0% real-world accuracy) - Identify data format integration needs (rtl_433 JSON, URH, GPX, etc.) - Share their wardriving workflows and preferences Document includes: - What's working: Device ID system, protocol database, GPS privacy features - Critical gaps: Real-world performance analysis (REAL_CAPTURE_ANALYSIS.md findings) - Data format questions: Hardware, file formats, GPS association methods, batch uploads - Integration roadmap: Prioritized format support based on community feedback - Response template: Copy/paste survey for wardrivers to share their setup Purpose: Gather requirements from experienced RF community to build seamless upload workflows for existing wardriving tools (RTL-SDR, HackRF, rtl_433, URH, etc.) beyond just Flipper Zero .sub files. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
leetcrypt
+286
-425
@@ -1,495 +1,356 @@
|
||||
# GigLez: Wardriver Onboarding Brief
|
||||
# GigLez - RF Wardriver Onboarding & Data Format Integration
|
||||
|
||||
**TL;DR:** We're building Wigle.net for Sub-GHz IoT devices. Upload RF captures (.sub files) + GPS → auto-identify devices → map IoT infrastructure globally.
|
||||
**Welcome, RF Wardrivers!** This document explains what we've built, current system capabilities, and **critically, what data formats we need to support for your existing wardriving workflows**.
|
||||
|
||||
---
|
||||
|
||||
## What is GigLez?
|
||||
## What We've Built: GigLez Overview
|
||||
|
||||
**Crowdsourced RF IoT Device Mapping Platform**
|
||||
**GigLez** is a Wigle-style crowdsourced platform for mapping Sub-GHz IoT RF devices (300-928 MHz). Think "Wigle.net but for garage doors, weather stations, tire pressure monitors, and doorbells instead of WiFi."
|
||||
|
||||
- **Like Wigle:** Users upload captures → database grows → community maps infrastructure
|
||||
- **Unlike Wigle:** Instead of WiFi/BT (2.4GHz), we focus on **Sub-GHz IoT** (315/433/868/915 MHz)
|
||||
- **Devices:** Weather sensors, garage openers, tire pressure monitors, doorbells, security sensors
|
||||
### Core Concept
|
||||
- **Upload** .sub files (Flipper Zero format) with GPS coordinates
|
||||
- **Automatic Device Identification** using RF signature matching (299 protocol database)
|
||||
- **Interactive Map** showing IoT device distribution by location
|
||||
- **Community Verification** for device IDs and crowdsourced improvements
|
||||
|
||||
---
|
||||
|
||||
## Why Sub-GHz?
|
||||
## What's Working Right Now
|
||||
|
||||
**Massive blind spot in IoT security/research:**
|
||||
- 433 MHz = most popular IoT frequency (weather, remotes, sensors)
|
||||
- No centralized database like Wigle (WiFi) or Shodan (internet)
|
||||
- Devices broadcast constantly, no encryption, easy to capture
|
||||
- Security implications: replay attacks, device tracking, privacy leaks
|
||||
### ✅ Device Identification System (6-Component Scoring)
|
||||
|
||||
**Your wardriving experience transfers perfectly:**
|
||||
- GPS logging → same workflow
|
||||
- Signal capture → Flipper Zero instead of WiFi adapter
|
||||
- Upload interface → familiar Wigle-style submission
|
||||
- Mapping → identical visualization approach
|
||||
We've built a multi-factor RF signature matching engine with **33% accuracy on synthetic signals** (baseline established):
|
||||
|
||||
---
|
||||
| Component | Weight | Purpose |
|
||||
|-----------|--------|---------|
|
||||
| **Timing Analysis** | 40% | K-means clustering to extract SHORT/LONG pulse widths |
|
||||
| **Preamble Detection** | 25% | Identifies sync patterns (alternating, long_burst, sync_word) |
|
||||
| **Timing Ratio** | 20% | Discriminates protocols by LONG/SHORT ratio (2:1 vs 3:1) |
|
||||
| **Frequency Fingerprinting** | 10% | ISM band classification (315/433/868/915 MHz, ±100kHz tolerance) |
|
||||
| **Bit Count Matching** | 5% | Validates bit length against protocol expectations |
|
||||
|
||||
## Current System Architecture
|
||||
**Processing Speed**: 156ms average per signal (0.4ms parse + 156ms matching)
|
||||
|
||||
```
|
||||
User uploads .sub file + GPS
|
||||
↓
|
||||
Parse RF Signal
|
||||
(frequency, pulses, timing)
|
||||
↓
|
||||
Multi-Decoder Pipeline:
|
||||
1. RTL_433 (200+ protocols)
|
||||
2. Pattern Decoder (timing analysis)
|
||||
↓
|
||||
Device Identified
|
||||
(LaCrosse TX141, Acurite 5n1, etc.)
|
||||
↓
|
||||
Store in PostGIS Database
|
||||
↓
|
||||
Display on Interactive Map
|
||||
```
|
||||
**Supported Modulations**: OOK, FSK (from .sub metadata)
|
||||
|
||||
---
|
||||
### ✅ Protocol Database (299 Protocols)
|
||||
|
||||
## Device Identification Algorithm (Current)
|
||||
Imported from RTL_433 project:
|
||||
- **Weather Sensors**: LaCrosse, Oregon Scientific, Acurite, Nexus (70+ protocols)
|
||||
- **Garage Doors**: Chamberlain, LiftMaster, Linear (15+ protocols)
|
||||
- **TPMS**: Schrader, Toyota, Honda, Ford (30+ protocols)
|
||||
- **Security**: Honeywell, SimpliSafe, DSC (20+ protocols)
|
||||
- **Doorbells & Remotes**: Princeton, PT2262, EV1527 (40+ protocols)
|
||||
- **Lighting/Climate**: LED controllers, ceiling fans, thermostats (20+ protocols)
|
||||
|
||||
### Decoder 1: RTL_433 Integration
|
||||
**What:** Subprocess wrapper around RTL_433 binary (FOSS, 15+ years development)
|
||||
**Protocols:** 286 devices (weather, automotive, security)
|
||||
**Method:**
|
||||
- Converts .sub → RTL_433 pulse format
|
||||
- Runs protocol matchers (one per device type)
|
||||
- Returns JSON if decoded
|
||||
**Frequency Coverage**:
|
||||
- 315 MHz: 15% (garage doors, car keys)
|
||||
- 433 MHz: 70% (weather, doorbells, generic remotes)
|
||||
- 868 MHz: 10% (EU devices)
|
||||
- 915 MHz: 5% (US ISM band)
|
||||
|
||||
**Performance:**
|
||||
- Speed: <100ms
|
||||
- Accuracy: 95% for known protocols
|
||||
- Limitation: Requires multi-transmission captures (RTL_433 expects repetitions)
|
||||
### ✅ GPS Validation & Privacy Features
|
||||
|
||||
**Location:** `src/matcher/rtl433_decoder.py`
|
||||
**Implemented** (`src/gps/validator.py`):
|
||||
- Latitude/longitude validation (-90 to 90, -180 to 180)
|
||||
- Coordinate precision rounding (configurable 4-8 decimal places)
|
||||
- Altitude validation (optional, -500m to 10000m)
|
||||
- Haversine distance calculation for duplicate detection
|
||||
- **Privacy**: GPS anonymization via coordinate rounding
|
||||
|
||||
### Decoder 2: Pattern-Based Heuristics
|
||||
**What:** Custom timing pattern analyzer for single-transmission captures
|
||||
**Method:**
|
||||
1. **Timing Extraction:** K-means cluster pulse widths → SHORT/LONG identification
|
||||
2. **Binary Decoding:** Convert pulses to bits (SHORT=0, LONG=1 for PWM)
|
||||
3. **Statistical Fingerprint:** Calculate mean pulse, duty cycle, pulse-gap ratio, pulse count
|
||||
4. **Database Matching:** Compare against 25+ protocol signatures
|
||||
5. **Confidence Scoring:** Weighted by timing accuracy (40%), bit count (30%), pattern match (30%)
|
||||
|
||||
**Performance:**
|
||||
- Speed: <50ms
|
||||
- Accuracy: 65% for protocol DB, 5% for unknowns
|
||||
- Works on: Flipper Zero short captures (1 button press)
|
||||
|
||||
**Location:** `src/matcher/pattern_decoder.py`
|
||||
|
||||
---
|
||||
|
||||
## Open-Source Resources Available
|
||||
|
||||
### 1. RTL_433 Protocol Database
|
||||
**Source:** https://github.com/merbanan/rtl_433
|
||||
- **286 device protocols** with timing signatures
|
||||
- **JSON export:** `data/rtl_433_protocols.json`
|
||||
- **Fields:** short_width, long_width, gap_limit, reset_limit, modulation
|
||||
- **Categories:** Weather (majority), automotive TPMS, security, doorbells
|
||||
|
||||
### 2. Flipper Zero .sub File Collections
|
||||
**Source:** Zero-Sploit/FlipperZero-Subghz-DB (13,717 files)
|
||||
- Community-contributed signal captures
|
||||
- Organized by device type
|
||||
- RAW pulse data + metadata
|
||||
- **Limitation:** Most labeled by remote function, not device model
|
||||
|
||||
### 3. FCC Equipment Authorization Database
|
||||
**Source:** https://fccid.io/
|
||||
- **All RF devices sold in US** must be certified
|
||||
- Contains: Operating frequency, power, device photos, manuals
|
||||
- **Use case:** Cross-reference identified devices, validate frequency ranges
|
||||
- **API:** Available for bulk lookups
|
||||
|
||||
### 4. GigLez Protocol Database
|
||||
**Source:** `src/matcher/protocol_database.py`
|
||||
- **25 hand-curated protocols** extracted from Flipper firmware + RTL_433
|
||||
- **Detailed timing:** Short/long pulse widths, preamble patterns, sync words
|
||||
- **Categories:** Weather (7), garage openers (3), doorbells (1), TPMS (2), security (1), remotes (6)
|
||||
|
||||
---
|
||||
|
||||
## Proposed Algorithm Improvements
|
||||
|
||||
### Problem 1: Low Identification Rate for Unknown Devices
|
||||
**Current:** 5% accuracy on devices not in protocol database
|
||||
**Impact:** Most user uploads return "Unknown"
|
||||
|
||||
### Problem 2: Single-Transmission Weakness
|
||||
**Current:** RTL_433 needs repetitions, pattern decoder struggles with noise
|
||||
**Impact:** Flipper captures (1 button press) often fail
|
||||
|
||||
### Problem 3: No Learning from User Feedback
|
||||
**Current:** System static, doesn't improve over time
|
||||
**Impact:** Missed opportunity to crowd-source knowledge
|
||||
|
||||
---
|
||||
|
||||
## Improved Heuristic Algorithm (FOSS-Only)
|
||||
|
||||
### Enhancement 1: Multi-Pass Timing Analysis
|
||||
|
||||
**Current Approach:**
|
||||
**Example**:
|
||||
```python
|
||||
# Single K-means clustering on all pulses
|
||||
pulses = [520, 1040, 480, 1020, ...]
|
||||
short, long = kmeans(pulses, k=2) # Assumes 2 distinct widths
|
||||
from src.gps.validator import validate_coordinates, anonymize_location
|
||||
|
||||
# Validate raw GPS
|
||||
coords = validate_coordinates(lat=40.712776, lon=-74.005974, altitude=10.5)
|
||||
|
||||
# Anonymize (round to ~11m precision)
|
||||
anon = anonymize_location(lat=40.712776, lon=-74.005974, precision=4)
|
||||
# Returns: (40.7128, -74.0060)
|
||||
```
|
||||
|
||||
**Improved Approach:**
|
||||
```python
|
||||
# Hierarchical clustering + outlier removal
|
||||
def extract_timing_robust(pulses):
|
||||
# Step 1: Remove outliers (noise, glitches)
|
||||
pulses_clean = remove_outliers(pulses, method='IQR')
|
||||
### ✅ File Format Parsing (.sub files)
|
||||
|
||||
# Step 2: Separate HIGH vs LOW pulses
|
||||
high_pulses = [p for p in pulses if p > 0]
|
||||
low_pulses = [abs(p) for p in pulses if p < 0]
|
||||
**Currently Supported**: Flipper Zero .sub format (both RAW and decoded variants)
|
||||
|
||||
# Step 3: Multi-level clustering
|
||||
# Try k=2,3,4 (some protocols have SHORT/MID/LONG)
|
||||
for k in [2, 3, 4]:
|
||||
clusters = kmeans(high_pulses, k=k)
|
||||
if is_valid_clustering(clusters): # Check separation
|
||||
return clusters
|
||||
**Parser** (`src/parser/sub_parser.py`):
|
||||
- Extracts frequency, preset (modulation), protocol name
|
||||
- Parses RAW_Data into pulse sequences
|
||||
- Handles both OOK and FSK modulation metadata
|
||||
- Validates file structure
|
||||
|
||||
# Step 4: Frequency histogram method (fallback)
|
||||
return histogram_peaks(high_pulses)
|
||||
**Example .sub file**:
|
||||
```
|
||||
Filetype: Flipper SubGhz RAW File
|
||||
Version: 1
|
||||
Frequency: 433920000
|
||||
Preset: FuriHalSubGhzPresetOok650Async
|
||||
Protocol: RAW
|
||||
RAW_Data: 2980 -240 520 -980 520 -980 1000 -500 ...
|
||||
```
|
||||
|
||||
**Benefit:** Handles multi-level modulation (e.g., tri-bit encoding)
|
||||
### ⚠️ Test Coverage
|
||||
|
||||
### Enhancement 2: Frequency-Based Protocol Filtering
|
||||
|
||||
**Current:** Search all 286 RTL_433 protocols
|
||||
**Improved:** Pre-filter by frequency band
|
||||
|
||||
```python
|
||||
FREQUENCY_PROTOCOL_MAP = {
|
||||
433920000: { # 433.92 MHz ISM band
|
||||
'weather': [12, 19, 20, 32, 40, 55, 73, 113], # RTL_433 protocol IDs
|
||||
'garage': [1, 8, 9],
|
||||
'security': [25, 26],
|
||||
},
|
||||
315000000: { # 315 MHz (North America)
|
||||
'automotive': [10, 11, 40], # TPMS
|
||||
'garage': [22, 23],
|
||||
},
|
||||
868000000: { # 868 MHz SRD (Europe)
|
||||
'weather': [78, 88, 113],
|
||||
'home_automation': [95, 102],
|
||||
}
|
||||
}
|
||||
|
||||
def filter_protocols_by_frequency(freq, tolerance=100_000):
|
||||
"""Return likely protocol IDs based on frequency"""
|
||||
freq_band = round_to_nearest_band(freq)
|
||||
return FREQUENCY_PROTOCOL_MAP.get(freq_band, [])
|
||||
```
|
||||
|
||||
**Benefit:** 10x speedup (test 20 protocols instead of 200)
|
||||
|
||||
### Enhancement 3: Preamble/Sync Pattern Detection
|
||||
|
||||
**Current:** Only checks if preamble exists in decoded bits
|
||||
**Improved:** Dedicated preamble detector before decoding
|
||||
|
||||
```python
|
||||
def detect_preamble(pulses):
|
||||
"""
|
||||
Preambles are repeating patterns at start of transmission
|
||||
Examples:
|
||||
- Oregon Scientific: 16x "10" = 32 alternating pulses
|
||||
- Princeton: 4x "1111" = long HIGH burst
|
||||
"""
|
||||
# Check first 50 pulses for repetition
|
||||
first_50 = pulses[:50]
|
||||
|
||||
# Method 1: Autocorrelation for periodic patterns
|
||||
period = find_autocorrelation_peak(first_50)
|
||||
if period:
|
||||
pattern = first_50[:period]
|
||||
repetitions = count_repetitions(first_50, pattern)
|
||||
if repetitions >= 4:
|
||||
return {
|
||||
'type': 'periodic',
|
||||
'pattern_length': period,
|
||||
'repetitions': repetitions
|
||||
}
|
||||
|
||||
# Method 2: Long burst detection (e.g., "1111...")
|
||||
if first_50[0] > mean(first_50) * 2: # First pulse much longer
|
||||
return {'type': 'long_burst', 'duration': first_50[0]}
|
||||
|
||||
return None
|
||||
```
|
||||
|
||||
**Benefit:** Narrow down protocols before full decode (faster + more accurate)
|
||||
|
||||
### Enhancement 4: Protocol Signature Expansion
|
||||
|
||||
**Current:** 25 protocols in `protocol_database.py`
|
||||
**Target:** Expand to 100+ using RTL_433 JSON
|
||||
|
||||
**Automated Extraction Script:**
|
||||
```python
|
||||
def import_rtl433_protocols():
|
||||
"""
|
||||
Parse RTL_433 source code to extract timing signatures
|
||||
|
||||
RTL_433 C code format:
|
||||
.short_width = 500,
|
||||
.long_width = 1000,
|
||||
.gap_limit = 2000,
|
||||
.reset_limit = 5000,
|
||||
"""
|
||||
rtl433_repo = "~/rtl_433/src/devices/"
|
||||
protocols = []
|
||||
|
||||
for c_file in glob(f"{rtl433_repo}/*.c"):
|
||||
# Regex extraction from C structs
|
||||
signature = extract_timing_from_c(c_file)
|
||||
if signature:
|
||||
protocols.append(ProtocolSignature(
|
||||
name=signature['name'],
|
||||
short_pulse_us=signature['short_width'],
|
||||
long_pulse_us=signature['long_width'],
|
||||
# ... more fields
|
||||
))
|
||||
|
||||
return protocols
|
||||
```
|
||||
|
||||
**Benefit:** 4x larger protocol database (25 → 100+), no manual curation
|
||||
|
||||
### Enhancement 5: Device Disambiguation via Metadata
|
||||
|
||||
**Problem:** Multiple devices have identical timing (e.g., Princeton = generic chipset)
|
||||
**Solution:** Use secondary characteristics
|
||||
|
||||
```python
|
||||
def disambiguate_matches(matches, metadata):
|
||||
"""
|
||||
Rank matches using:
|
||||
1. Frequency exact match (higher weight)
|
||||
2. Bit count exact match
|
||||
3. Preamble pattern match
|
||||
4. Geographic prior (common devices in region)
|
||||
"""
|
||||
scored = []
|
||||
for match in matches:
|
||||
score = match.confidence
|
||||
|
||||
# Bonus: Exact frequency match
|
||||
if abs(match.frequency - metadata.frequency) < 10_000:
|
||||
score *= 1.2
|
||||
|
||||
# Bonus: Bit count perfect match
|
||||
bit_count = len(metadata.decoded_bits)
|
||||
if match.min_bits <= bit_count <= match.max_bits:
|
||||
if bit_count == match.typical_bits:
|
||||
score *= 1.15
|
||||
|
||||
# Bonus: Preamble detected and matches
|
||||
if metadata.preamble and match.preamble_pattern:
|
||||
if metadata.preamble.startswith(match.preamble_pattern):
|
||||
score *= 1.3
|
||||
|
||||
# Bonus: Common in user's region (from GPS)
|
||||
if metadata.gps:
|
||||
regional_devices = get_common_devices_nearby(metadata.gps)
|
||||
if match.name in regional_devices:
|
||||
score *= 1.1
|
||||
|
||||
scored.append((match, score))
|
||||
|
||||
return sorted(scored, key=lambda x: x[1], reverse=True)
|
||||
```
|
||||
|
||||
**Benefit:** Princeton @ 433MHz + 24-bit → could be garage opener OR remote → GPS (residential area) → likely garage opener
|
||||
- **Unit Tests**: 56 tests passing (timing, preamble, frequency, GPS, database)
|
||||
- **Synthetic Benchmark**: 33% top-1 accuracy on 12 test signals
|
||||
- **Real-World Benchmark**: **0% accuracy on 11 real Flipper captures** ❌
|
||||
|
||||
---
|
||||
|
||||
## Data Sources for Protocol Expansion
|
||||
## Critical Gap: Real-World Performance Issue
|
||||
|
||||
### Source 1: RTL_433 Device C Files
|
||||
**Path:** https://github.com/merbanan/rtl_433/tree/master/src/devices
|
||||
**Count:** 286 .c files
|
||||
**Extractable Data:**
|
||||
- Timing parameters (short/long/gap/reset widths)
|
||||
- Modulation type (OOK/FSK)
|
||||
- Bit lengths
|
||||
- Manufacturer/model names
|
||||
### The Problem: Synthetic vs Real
|
||||
|
||||
**Extraction Method:** Regex parsing of C structs
|
||||
We optimized for synthetic test signals, but **real Flipper Zero community captures fail completely**.
|
||||
|
||||
### Source 2: Flipper Zero Firmware
|
||||
**Path:** https://github.com/flipperdevices/flipperzero-firmware/tree/dev/lib/subghz/protocols
|
||||
**Count:** ~40 protocol decoders
|
||||
**Extractable Data:**
|
||||
- Timing tolerances
|
||||
- Encoding schemes (PWM, Manchester, etc.)
|
||||
- Preamble patterns
|
||||
- Sample data payloads
|
||||
**Root Causes** (analyzed in `docs/REAL_CAPTURE_ANALYSIS.md`):
|
||||
|
||||
**Extraction Method:** Parse C protocol definitions
|
||||
1. **Decoded vs RAW Format** (45% of failures)
|
||||
- Many real .sub files are **already decoded** (`Protocol: Holtek_HT12X`)
|
||||
- Our system only processes `RAW_Data` fields
|
||||
- **Fix Needed**: Parse decoded .sub files by protocol name + timing element (TE)
|
||||
|
||||
### Source 3: Universal Radio Hacker (URH)
|
||||
**Path:** https://github.com/jopohl/urh
|
||||
**Tool:** GUI for reverse-engineering RF protocols
|
||||
**Output:** XML protocol definitions
|
||||
**Use Case:** Community could contribute URH-analyzed protocols
|
||||
2. **Missing Protocols** (27% of failures)
|
||||
- Database lacks: Acurite 02077M, GE Doorbell 19297, Byron DB421E, Holtek HT12X
|
||||
- **Fix Needed**: Import from real captures or community contributions
|
||||
|
||||
### Source 4: GigLez User Submissions
|
||||
**Method:** Crowd-source unknown signals
|
||||
**Workflow:**
|
||||
1. User uploads "Unknown" capture
|
||||
2. Admin/community analyzes with URH or manual tools
|
||||
3. Creates protocol signature
|
||||
4. Adds to database → future captures auto-matched
|
||||
3. **Signal Complexity** (27% of failures)
|
||||
- Real captures have multi-packet transmissions (131 pulses vs 40 bits)
|
||||
- Variable signal quality, interference beyond 15% jitter tolerance
|
||||
- **Fix Needed**: Packet segmentation, higher noise tolerance (15% → 25%)
|
||||
|
||||
**Gamification:** Leaderboard for protocol contributors (like Wigle)
|
||||
4. **Protocol Family Competition** (9% of failures)
|
||||
- Nexus ranks #34, beaten by similar Oregon Scientific protocols
|
||||
- **Fix Needed**: Protocol family grouping and boosting
|
||||
|
||||
---
|
||||
|
||||
## Implementation Priority
|
||||
## What We Need From You: Data Format Questions
|
||||
|
||||
### Phase 1: Protocol Database Expansion (Week 1)
|
||||
- [ ] Parse RTL_433 JSON → extract 100+ additional signatures
|
||||
- [ ] Import to `protocol_database.py`
|
||||
- [ ] Test: Does this improve accuracy on test dataset?
|
||||
To make GigLez seamless for **existing RF wardriving workflows**, we need to understand your data collection methods and formats.
|
||||
|
||||
### Phase 2: Improved Timing Analysis (Week 2)
|
||||
- [ ] Implement robust clustering with outlier removal
|
||||
- [ ] Add multi-level clustering (k=2,3,4)
|
||||
- [ ] Preamble detection algorithm
|
||||
- [ ] Benchmark: Accuracy on single-transmission captures
|
||||
### 1. Hardware & Capture Tools
|
||||
|
||||
### Phase 3: Frequency-Based Filtering (Week 3)
|
||||
- [ ] Build frequency → protocol ID mapping
|
||||
- [ ] Integrate with RTL_433 decoder (pass -R flags)
|
||||
- [ ] Benchmark: Speed improvement
|
||||
**What RF capture hardware do you use?**
|
||||
- [ ] Flipper Zero
|
||||
- [ ] RTL-SDR dongles
|
||||
- [ ] HackRF One
|
||||
- [ ] LimeSDR
|
||||
- [ ] YARD Stick One
|
||||
- [ ] LilyGo T-Watch / LoRa devices
|
||||
- [ ] Other: _______________
|
||||
|
||||
### Phase 4: Disambiguation Logic (Week 4)
|
||||
- [ ] Implement metadata-based scoring
|
||||
- [ ] Add geographic priors (query PostGIS for nearby device types)
|
||||
- [ ] Test: Reduction in ambiguous results
|
||||
**What software do you use to capture signals?**
|
||||
- [ ] Flipper Zero firmware (generates .sub files)
|
||||
- [ ] rtl_433 (outputs JSON/CSV)
|
||||
- [ ] Universal Radio Hacker (URH) - saves .complex/.cfile
|
||||
- [ ] GNU Radio (custom flowgraphs)
|
||||
- [ ] inspectrum (visual analysis)
|
||||
- [ ] SDR# (saves .wav / IQ recordings)
|
||||
- [ ] gqrx (IQ recordings)
|
||||
- [ ] Other: _______________
|
||||
|
||||
### 2. File Format Preferences
|
||||
|
||||
**What file formats do your captures produce?**
|
||||
- [ ] .sub (Flipper Zero)
|
||||
- [ ] .fff (Flipper Zero older format)
|
||||
- [ ] JSON (rtl_433 output)
|
||||
- [ ] CSV (rtl_433 CSV mode)
|
||||
- [ ] .complex / .cfile (URH raw IQ data)
|
||||
- [ ] .wav (SDR# IQ recordings)
|
||||
- [ ] .sigmf (SigMF metadata format)
|
||||
- [ ] .cu8 / .cs8 (raw IQ samples)
|
||||
- [ ] Other: _______________
|
||||
|
||||
**Would you prefer to upload:**
|
||||
- [ ] Individual files with GPS metadata embedded
|
||||
- [ ] Batch uploads (ZIP with manifest.json)
|
||||
- [ ] Real-time streaming (live wardriving session)
|
||||
- [ ] CSV/JSON logs referencing stored signal files
|
||||
|
||||
### 3. GPS Data Association
|
||||
|
||||
**How do you currently associate GPS coordinates with captures?**
|
||||
- [ ] Manual entry (lat/lon typed per file)
|
||||
- [ ] GPX track log synced by timestamp
|
||||
- [ ] NMEA GPS logs (separate file)
|
||||
- [ ] Embedded in capture file metadata
|
||||
- [ ] Kismet-style CSV (BSSID, lat, lon, timestamp)
|
||||
- [ ] Custom database/spreadsheet
|
||||
- [ ] Other: _______________
|
||||
|
||||
**GPS Precision Requirements:**
|
||||
- What precision do you typically capture? (e.g., 6 decimal places = ±0.11m)
|
||||
- Do you need altitude data stored?
|
||||
- Do you want GPS anonymization options (round to city-level precision)?
|
||||
|
||||
### 4. Metadata & Session Context
|
||||
|
||||
**What additional metadata do you typically log?**
|
||||
- [ ] Device photos (visual identification)
|
||||
- [ ] Signal strength (RSSI)
|
||||
- [ ] Capture duration
|
||||
- [ ] Antenna type/gain
|
||||
- [ ] Weather conditions
|
||||
- [ ] Session notes/comments
|
||||
- [ ] Device orientation (N/S/E/W)
|
||||
- [ ] Other: _______________
|
||||
|
||||
### 5. Batch Upload Workflows
|
||||
|
||||
**Typical wardriving session size:**
|
||||
- How many captures per session? (e.g., 50, 500, 5000+)
|
||||
- Do you prefer uploading during the drive or post-processing later?
|
||||
- Would you use a CLI tool for bulk uploads or web interface?
|
||||
|
||||
**Preferred batch format:**
|
||||
```
|
||||
Option A: ZIP with manifest.json
|
||||
captures.zip
|
||||
├── capture_001.sub
|
||||
├── capture_002.sub
|
||||
├── ...
|
||||
└── manifest.json # GPS + timestamps for all files
|
||||
|
||||
Option B: CSV index + separate signal files
|
||||
session_2025-01-15.csv (columns: filename, lat, lon, timestamp, notes)
|
||||
signals/
|
||||
├── capture_001.sub
|
||||
├── capture_002.sub
|
||||
└── ...
|
||||
|
||||
Option C: Real-time API streaming (JSON payloads)
|
||||
```
|
||||
|
||||
Which format would integrate best with your existing tools?
|
||||
|
||||
### 6. Device Identification Preferences
|
||||
|
||||
**When you capture unknown signals, what info helps you ID them?**
|
||||
- [ ] Frequency alone (e.g., "433.92 MHz probably garage door")
|
||||
- [ ] Timing patterns (pulse widths)
|
||||
- [ ] Bit patterns (binary data)
|
||||
- [ ] Physical device photo
|
||||
- [ ] FCC ID database lookup
|
||||
- [ ] Community voting/verification
|
||||
- [ ] Other: _______________
|
||||
|
||||
**Confidence threshold for auto-tagging:**
|
||||
- Would you trust automatic IDs at 80% confidence?
|
||||
- Would you manually verify all IDs regardless of confidence?
|
||||
- Prefer "suggested IDs" that you confirm before publishing?
|
||||
|
||||
### 7. Privacy & Data Sharing
|
||||
|
||||
**Data sharing preferences:**
|
||||
- [ ] Make all captures public (Wigle-style open database)
|
||||
- [ ] Private by default (opt-in sharing)
|
||||
- [ ] Share anonymized location data only (round to city)
|
||||
- [ ] Share device types but not GPS coordinates
|
||||
- [ ] Custom privacy controls per capture
|
||||
|
||||
**Removal requests:**
|
||||
- Should users be able to request removal of specific captures (like Wigle's BSSID opt-out)?
|
||||
|
||||
---
|
||||
|
||||
## Expected Improvements
|
||||
## Integration Roadmap
|
||||
|
||||
| Metric | Current | Target | Method |
|
||||
|--------|---------|--------|--------|
|
||||
| **Protocol DB Size** | 25 | 100+ | Auto-extract RTL_433 |
|
||||
| **Unknown Device Accuracy** | 5% | 40% | Expanded DB + robust timing |
|
||||
| **Single-Tx Success Rate** | 20% | 65% | Outlier removal + preamble detection |
|
||||
| **Disambiguation Accuracy** | 60% | 85% | Metadata scoring |
|
||||
| **Avg Processing Time** | 150ms | 80ms | Frequency filtering |
|
||||
Based on your feedback, we'll prioritize:
|
||||
|
||||
### Phase 1: Core Format Support
|
||||
- [ ] rtl_433 JSON import (most common for non-Flipper wardrivers)
|
||||
- [ ] Decoded .sub file support (fix 45% of current failures)
|
||||
- [ ] GPX track log parsing
|
||||
|
||||
### Phase 2: Batch Upload System
|
||||
- [ ] ZIP + manifest.json uploader
|
||||
- [ ] CLI tool for bulk submissions
|
||||
- [ ] API endpoints for real-time streaming
|
||||
|
||||
### Phase 3: Additional Hardware Support
|
||||
- [ ] URH .complex/.cfile import (convert to timing patterns)
|
||||
- [ ] SDR# .wav IQ file processing
|
||||
- [ ] SigMF metadata parsing
|
||||
|
||||
### Phase 4: Community Features
|
||||
- [ ] Manual device ID submission
|
||||
- [ ] Photo upload for visual verification
|
||||
- [ ] Voting system for ID accuracy
|
||||
- [ ] Protocol signature contributions
|
||||
|
||||
---
|
||||
|
||||
## How You Can Contribute
|
||||
## How to Contribute Your Data Now
|
||||
|
||||
### As Wardriver with Data:
|
||||
1. **Upload Captures:** If you have Flipper Zero, start wardrive-style captures
|
||||
- Walk/drive with Flipper in "Read RAW" mode
|
||||
- Save .sub files with GPS timestamps
|
||||
- Bulk upload via API or web interface
|
||||
### Option 1: Share Sample Files
|
||||
Send us 5-10 representative captures from your wardriving sessions:
|
||||
- Include GPS coordinates (lat/lon/timestamp)
|
||||
- Mix of known and unknown devices
|
||||
- Note your capture tool and workflow
|
||||
|
||||
2. **Verify Identifications:** Review auto-matched devices
|
||||
- Confirm: "Yes, that's a LaCrosse sensor"
|
||||
- Correct: "No, it's an Acurite 5n1"
|
||||
- Feedback trains future improvements
|
||||
**Where to send**: [Create GitHub issue](https://github.com/yourusername/giglez/issues) or email
|
||||
|
||||
3. **Map Coverage:** Apply Wigle strategy
|
||||
- Focus on under-mapped areas
|
||||
- Multiple passes for verification
|
||||
- Track unique devices vs observations
|
||||
|
||||
### As RF/Protocol Expert:
|
||||
1. **Protocol Analysis:** Help identify unknowns
|
||||
- Use Universal Radio Hacker (URH)
|
||||
- Document timing patterns
|
||||
- Submit to protocol database
|
||||
|
||||
2. **Algorithm Tuning:** Test decoder variants
|
||||
- Different clustering methods
|
||||
- Thresholds for confidence scoring
|
||||
- Edge cases (noise, interference)
|
||||
|
||||
3. **Dataset Creation:** Build labeled test set
|
||||
- Purchase 10-20 common devices
|
||||
- Capture ground-truth signals
|
||||
- Use for accuracy benchmarking
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
||||
### View Current System:
|
||||
### Option 2: Test the Current System
|
||||
Upload Flipper Zero .sub files via our API (coming soon):
|
||||
```bash
|
||||
cd /home/dell/coding/giglez
|
||||
|
||||
# See protocol database
|
||||
python src/matcher/protocol_database.py
|
||||
|
||||
# Test pattern decoder
|
||||
python src/matcher/pattern_decoder.py
|
||||
|
||||
# Check RTL_433 integration
|
||||
python src/matcher/rtl433_decoder.py
|
||||
curl -X POST https://giglez.io/api/submit \
|
||||
-F "file=@capture.sub" \
|
||||
-F "latitude=40.7128" \
|
||||
-F "longitude=-74.0060" \
|
||||
-F "timestamp=2025-01-15T12:00:00Z"
|
||||
```
|
||||
|
||||
### Test on Sample Data:
|
||||
### Option 3: Provide rtl_433 JSON
|
||||
If you use rtl_433, share sample JSON output:
|
||||
```bash
|
||||
# We have RTL_433 test captures
|
||||
ls data/rf_test_datasets/rtl_433_tests/tests/
|
||||
|
||||
# Example: Decode a weather sensor
|
||||
python scripts/test_rtl433_with_known_devices.py
|
||||
```
|
||||
|
||||
### Database Schema:
|
||||
```sql
|
||||
-- Key tables
|
||||
captures (id, lat, lon, timestamp, frequency, file_path, pulse_count)
|
||||
devices (id, name, manufacturer, category, frequency)
|
||||
identifications (capture_id, device_id, confidence, method)
|
||||
rtl_433 -F json > captures.json
|
||||
# Include GPS coordinates for each capture (manual or GPX sync)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Questions?
|
||||
|
||||
**Codebase:** `/home/dell/coding/giglez/`
|
||||
**Docs:**
|
||||
- `docs/ML_DESIGN.md` - Full ML research (ignore if focusing on heuristics)
|
||||
- `CLAUDE.md` - Project overview
|
||||
- `docs/IMPLEMENTATION_SUMMARY.md` - Current progress
|
||||
We want to build this **for the wardriving community**, so your input directly shapes development priorities.
|
||||
|
||||
**Similar Projects:**
|
||||
- Wigle.net (WiFi wardriving) - our inspiration
|
||||
- RTL_433 (device decoder) - our decoder backend
|
||||
- Flipper Zero (capture tool) - our data source
|
||||
|
||||
**Next Steps:**
|
||||
1. Review protocol database expansion script (Phase 1)
|
||||
2. Discuss disambiguation heuristics (Phase 4)
|
||||
3. Define accuracy benchmarks for success criteria
|
||||
**Contact**:
|
||||
- GitHub Issues: [giglez/issues](https://github.com/yourusername/giglez/issues)
|
||||
- Email: team@giglez.io (placeholder)
|
||||
- Discord: [GigLez Community](https://discord.gg/giglez) (placeholder)
|
||||
|
||||
---
|
||||
|
||||
**Document Version:** 1.0
|
||||
**Last Updated:** 2026-02-14
|
||||
**Status:** Ready for collaborator review
|
||||
## Response Template (Copy/Paste & Fill Out)
|
||||
|
||||
```
|
||||
### My Wardriving Setup
|
||||
|
||||
**Hardware**: [e.g., HackRF One + GPS dongle]
|
||||
**Software**: [e.g., rtl_433 + GPX logger]
|
||||
**File Formats**: [e.g., JSON output from rtl_433]
|
||||
**GPS Method**: [e.g., GPX track synced by timestamp]
|
||||
**Session Size**: [e.g., 200-500 captures per drive]
|
||||
**Upload Preference**: [e.g., Batch upload via CLI after session]
|
||||
|
||||
### Data Format Details
|
||||
|
||||
[Paste sample file snippet or describe structure]
|
||||
|
||||
### Integration Requests
|
||||
|
||||
[What would make GigLez fit seamlessly into your workflow?]
|
||||
|
||||
### Privacy Preferences
|
||||
|
||||
[Public/private/anonymized?]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**Thank you for helping build the Sub-GHz IoT mapping platform!** 🚗📡
|
||||
|
||||
*Last Updated: 2026-02-16*
|
||||
|
||||
Reference in New Issue
Block a user