# GigLez - IoT RF Device Mapping Platform ## Primary Directive Build a Wigle-like crowdsourced platform for mapping Sub-GHz RF IoT devices. Accept .sub/.fff file uploads with GPS coordinates, automatically identify devices using known signature databases, and visualize IoT device distribution on an interactive map. ## Core Objectives ### 1. Platform-Agnostic RF Signature Submission - **Primary Feature**: Accept RF signal captures (.sub, .fff files) with GPS coordinates from ANY capture device - **Submission Requirements**: - GPS coordinates (latitude/longitude) - REQUIRED - Timestamp - REQUIRED - .sub or .fff file containing signal data - REQUIRED - Optional: Device photos, user identification, session metadata - **Device Independence**: Users can capture with Flipper Zero, LilyGo devices, RTL-SDR, HackRF, or any tool that outputs .sub/.fff format ### 2. Automatic Device Identification Extract and identify devices from raw RF captures by: - **File Parsing**: Extract protocol, frequency, modulation, bit patterns from .sub/.fff files - **Database Matching**: Compare against known signature databases (Flipper Zero, RTL_433) - **Confidence Scoring**: Rank matches by similarity (exact, partial, pattern-based) - **Community Verification**: Allow users to confirm/correct automatic identifications ### 3. Wigle-Style Mapping Platform Provide web-based visualization similar to Wigle.net: - **Interactive Map**: Display captured devices with GPS markers - **Heatmap View**: Show device density by geographic area - **Search & Filter**: By device type, frequency, protocol, date range - **Statistics Dashboard**: Total captures, unique devices, geographic coverage - **Leaderboard**: Top contributors by uploads/verifications ## Wigle.net Analysis & Implementation Strategy ### Key Wigle.net Features to Replicate #### 1. Submission Workflow **Wigle Approach:** - CSV upload with standardized format - Required fields: MAC, SSID, GPS coords, timestamp - Pre-header with client metadata - Batch uploads supported **GigLez Implementation:** - Accept .sub/.fff files + GPS JSON/CSV - Required fields: GPS (lat/lon), timestamp, signal file - Upload via web interface or API - Support batch submissions (ZIP of .sub files + manifest.json) #### 2. Data Storage **Wigle Stats (2017):** - 349M WiFi networks - 7.8M cell towers - Billions of observations - GPS coordinates for 99%+ of records **GigLez Architecture:** - PostgreSQL with PostGIS for geospatial queries - Deduplicate by file hash + GPS proximity - Store both raw files and parsed metadata - Index by frequency, protocol, location, timestamp #### 3. Search & Discovery **Wigle Features:** - Text search (SSID, MAC) - Geographic search (bounding box, radius) - Advanced filters (encryption, date range) - Export to CSV/KML **GigLez Equivalent:** - Search by device type, manufacturer, protocol - Geographic search (radius, bounding box) - Filter by frequency, modulation, confidence score - Export to .sub, JSON, CSV, GeoJSON #### 4. Mapping Interface **Wigle UI:** - Zoom-based detail levels - Color coding by signal type/quality - Click for network details - Overlays from entire database **GigLez UI:** - Leaflet.js/Mapbox for mapping - Color by device type or frequency band - Marker clustering for performance - Click for device details (.sub file viewer) - Heatmap overlay for density #### 5. User Accounts & Gamification **Wigle System:** - User registration required - Upload tracking and statistics - Leaderboard (global/monthly) - Contribution milestones **GigLez System:** - Optional accounts (allow anonymous uploads) - Track uploads, identifications, verifications - Reputation score for accurate IDs - Badges for contributions (first capture in city, 100 devices, etc.) #### 6. API Access **Wigle API:** - JSON-based RPC (not REST) - Authentication via API token - Query endpoints for searching - Upload endpoints for submissions - Rate limiting **GigLez API:** - RESTful JSON API - JWT tokens + API keys - Endpoints: - `POST /api/submit` - Upload captures - `GET /api/search` - Query database - `GET /api/devices/{id}` - Device details - `GET /api/heatmap` - Density data - `GET /api/stats` - Platform statistics ### Differences from Wigle | Feature | Wigle.net | GigLez | |---------|-----------|--------| | **Data Type** | WiFi, Bluetooth, Cellular | Sub-GHz IoT RF (300-928 MHz) | | **Submission Format** | CSV | .sub/.fff files + GPS | | **Identification** | MAC/SSID (exact) | Protocol signature matching (fuzzy) | | **Focus** | Network mapping | Device type identification | | **Privacy** | Public by default | Opt-in sharing, anonymization | ## Technical Specifications ### Supported File Formats #### 1. Flipper Zero .sub Format ``` Filetype: Flipper SubGhz Key File Version: 1 Frequency: 433920000 Preset: FuriHalSubGhzPresetOok270Async Protocol: Princeton Bit: 24 Key: 00 00 00 00 00 95 D5 D4 TE: 400 ``` **Key Fields for Matching:** - `Frequency`: Exact frequency in Hz - `Preset`: Modulation type (OOK/FSK) - `Protocol`: Protocol name (if decoded) - `Bit`: Bit length - `Key`: Data payload (hex) - `TE`: Timing element (μs) #### 2. Flipper RAW Format (.sub) ``` Filetype: Flipper SubGhz RAW File Version: 1 Frequency: 433920000 Preset: FuriHalSubGhzPresetOok650Async Protocol: RAW RAW_Data: 2980 -240 520 -980 520 -980 ... ``` **Parsing Strategy:** - Extract timing patterns - Identify repeating sequences - Match against known protocols by timing signatures - Calculate pulse width statistics #### 3. .fff Format (Future Feature) Support for other firmware formats as needed. ### Submission API Format #### JSON Manifest ```json { "submission": { "timestamp": "2025-01-11T20:30:00Z", "latitude": 40.7128, "longitude": -74.0060, "altitude": 10.5, "accuracy": 5.0, "session_id": "optional_session_identifier", "user_id": "optional_user_identifier", "device_name": "Flipper Zero", "notes": "Captured near downtown" }, "files": [ { "filename": "capture_001.sub", "sha256": "abc123...", "size_bytes": 256 }, { "filename": "capture_002.sub", "sha256": "def456...", "size_bytes": 312 } ] } ``` #### CSV Submission Format (Alternative) ```csv latitude,longitude,timestamp,filename,accuracy,altitude 40.7128,-74.0060,2025-01-11T20:30:00Z,capture_001.sub,5.0,10.5 40.7129,-74.0061,2025-01-11T20:30:15Z,capture_002.sub,5.0,10.5 ``` ### Device Signature Matching Pipeline #### Step 1: Parse .sub File ```python def parse_sub_file(file_path): """Extract metadata from .sub file""" metadata = { 'frequency': None, 'protocol': None, 'modulation': None, 'bit_length': None, 'key_data': None, 'timing': None, 'raw_data': None } with open(file_path) as f: for line in f: if ':' in line: key, value = line.split(':', 1) # Map to metadata fields return metadata ``` #### Step 2: Match Against Signature Database ```python def match_signature(metadata): """ Match parsed metadata against known signatures Returns: [(device_id, confidence), ...] """ matches = [] # Exact match: protocol + frequency + bit length exact = query_exact_match( metadata['protocol'], metadata['frequency'], metadata['bit_length'] ) if exact: matches.append((exact.device_id, 1.0)) # Partial match: protocol + frequency partial = query_partial_match( metadata['protocol'], metadata['frequency'] ) for match in partial: matches.append((match.device_id, 0.8)) # Pattern match: bit pattern similarity if metadata['key_data']: pattern_matches = match_bit_patterns(metadata['key_data']) matches.extend(pattern_matches) # Timing match: for RAW files if metadata['raw_data']: timing_matches = match_timing_patterns(metadata['raw_data']) matches.extend(timing_matches) # Sort by confidence, deduplicate return sorted(set(matches), key=lambda x: x[1], reverse=True) ``` #### Step 3: Store Results ```python def store_capture(file_path, gps_coords, matches): """Store capture with matched device(s)""" capture = { 'latitude': gps_coords['lat'], 'longitude': gps_coords['lon'], 'timestamp': gps_coords['timestamp'], 'file_hash': sha256(file_path), 'file_path': upload_to_storage(file_path), **parse_sub_file(file_path) } # Store capture capture_id = db.captures.insert(capture) # Store top 3 matches for device_id, confidence in matches[:3]: db.capture_matches.insert({ 'capture_id': capture_id, 'device_id': device_id, 'confidence': confidence, 'method': 'auto' }) return capture_id ``` ## System Architecture ### Platform Components ``` ┌─────────────────────────────────────────────────────────┐ │ Web Interface │ │ - Upload Form (drag .sub files + GPS) │ │ - Interactive Map (Leaflet.js) │ │ - Search & Filter UI │ │ - Device Database Browser │ └─────────────────────────────────────────────────────────┘ │ ↓ ┌─────────────────────────────────────────────────────────┐ │ API Layer (FastAPI) │ │ POST /api/submit - Upload captures │ │ GET /api/search - Query database │ │ GET /api/devices - Device catalog │ │ GET /api/heatmap - Geographic density │ └─────────────────────────────────────────────────────────┘ │ ↓ ┌─────────────────────────────────────────────────────────┐ │ Processing Pipeline │ │ 1. File Parser (.sub/.fff → metadata) │ │ 2. Signature Matcher (metadata → device IDs) │ │ 3. Deduplicator (file hash + GPS proximity) │ │ 4. Storage Manager (DB + file storage) │ └─────────────────────────────────────────────────────────┘ │ ↓ ┌─────────────────────────────────────────────────────────┐ │ Database (PostgreSQL + PostGIS) │ │ - captures (GPS + metadata + file refs) │ │ - devices (known device types) │ │ - signatures (Flipper/RTL_433 patterns) │ │ - users (optional accounts) │ │ - identifications (community verifications) │ └─────────────────────────────────────────────────────────┘ │ ↓ ┌─────────────────────────────────────────────────────────┐ │ Signature Databases (Read-Only) │ │ - Flipper Zero .sub collection (1000+ files) │ │ - RTL_433 protocol definitions (200+ protocols) │ │ - Community-contributed signatures │ └─────────────────────────────────────────────────────────┘ ``` ### Key Features #### Core Platform Features - **File Upload**: Drag-and-drop .sub files with GPS coordinates - **Automatic Parsing**: Extract frequency, protocol, modulation, data from files - **Device Matching**: Identify devices using signature databases - **Deduplication**: Prevent duplicate submissions via file hashing - **Geospatial Search**: Find captures near location or within bounding box - **Heatmap Generation**: Visualize device density by geographic area - **Export Data**: Download captures as .sub, JSON, CSV, GeoJSON #### Community Features - **Manual Identification**: Users can add/correct device IDs - **Photo Uploads**: Visual evidence of physical devices - **Voting System**: Upvote/downvote identifications - **Verification**: High-confidence IDs become verified - **Contribution Tracking**: Statistics per user - **Leaderboard**: Top uploaders and verifiers #### Privacy Features - **Anonymous Uploads**: No account required - **GPS Precision Control**: Round coordinates to configurable precision - **Private Captures**: Opt-out of public database - **BSSID-style Removal**: Allow device signature removal requests ## Development Phases ### Phase 1: Foundation (Weeks 1-2) - [x] Database schema design - [ ] .sub file parser implementation - [ ] GPS coordinate validation - [ ] Basic file upload endpoint - [ ] Storage backend (local/S3) ### Phase 2: Signature Matching (Weeks 3-4) - [ ] Import Flipper Zero .sub database - [ ] Import RTL_433 protocol definitions - [ ] Build matching engine (exact/partial/pattern) - [ ] Confidence scoring algorithm - [ ] Match result storage ### Phase 3: Web Interface (Weeks 5-6) - [ ] Upload form with drag-and-drop - [ ] Map visualization (Leaflet.js) - [ ] Search and filter UI - [ ] Device detail pages - [ ] Statistics dashboard ### Phase 4: API & Integration (Weeks 7-8) - [ ] RESTful API endpoints - [ ] Authentication (JWT/API keys) - [ ] Rate limiting - [ ] OpenAPI documentation - [ ] Client libraries (Python, JS) ### Phase 5: Community Features (Weeks 9-10) - [ ] User accounts (optional) - [ ] Manual device identification - [ ] Photo upload and display - [ ] Voting system - [ ] Verification workflow ### Phase 6: Optimization (Weeks 11-12) - [ ] Database indexing and optimization - [ ] Caching layer (Redis) - [ ] CDN for file storage - [ ] Batch processing queue - [ ] Materialized view updates ## Success Metrics ### Platform Growth - Number of unique captures submitted - Geographic coverage (cities/countries) - Total .sub files processed - Database size (captures, devices) ### Community Engagement - Active users (uploaders + verifiers) - Manual identifications submitted - Verification votes cast - Photo evidence uploads ### Data Quality - Device identification accuracy (verified/total) - Average confidence score - Duplicate detection rate - Geographic precision distribution ### Technical Performance - Upload processing time (median) - Search query latency (p95) - Map render performance - API response times ## Technical Constraints ### File Processing - .sub file size limits (1MB max recommended) - Batch upload limits (100 files or 50MB per request) - Supported file formats (.sub initially, .fff future) - File parsing timeout (5 seconds per file) ### Geographic Data - GPS coordinate precision (6-8 decimal places) - Coordinate validation (valid lat/lon ranges) - Altitude optional (meters above sea level) - Accuracy metadata (horizontal accuracy in meters) ### Database Scalability - PostgreSQL with PostGIS for geospatial - Partitioning by date for large datasets - Index strategy for common queries - Materialized views for statistics ### Privacy & Compliance - GDPR-style data removal - Optional account system - GPS anonymization (configurable rounding) - No PII in .sub file metadata