Initial commit: Phase 1 & Phase 2 infrastructure complete

2026-01-12 11:21:17 -08:00
commit eb225771bc
53 changed files with 13645 additions and 0 deletions
@@ -0,0 +1,474 @@
+# GigLez - IoT RF Device Mapping Platform
+
+## Primary Directive
+
+Build a Wigle-like crowdsourced platform for mapping Sub-GHz RF IoT devices. Accept .sub/.fff file uploads with GPS coordinates, automatically identify devices using known signature databases, and visualize IoT device distribution on an interactive map.
+
+## Core Objectives
+
+### 1. Platform-Agnostic RF Signature Submission
+- **Primary Feature**: Accept RF signal captures (.sub, .fff files) with GPS coordinates from ANY capture device
+- **Submission Requirements**:
+  - GPS coordinates (latitude/longitude) - REQUIRED
+  - Timestamp - REQUIRED
+  - .sub or .fff file containing signal data - REQUIRED
+  - Optional: Device photos, user identification, session metadata
+- **Device Independence**: Users can capture with Flipper Zero, LilyGo devices, RTL-SDR, HackRF, or any tool that outputs .sub/.fff format
+
+### 2. Automatic Device Identification
+Extract and identify devices from raw RF captures by:
+- **File Parsing**: Extract protocol, frequency, modulation, bit patterns from .sub/.fff files
+- **Database Matching**: Compare against known signature databases (Flipper Zero, RTL_433)
+- **Confidence Scoring**: Rank matches by similarity (exact, partial, pattern-based)
+- **Community Verification**: Allow users to confirm/correct automatic identifications
+
+### 3. Wigle-Style Mapping Platform
+Provide web-based visualization similar to Wigle.net:
+- **Interactive Map**: Display captured devices with GPS markers
+- **Heatmap View**: Show device density by geographic area
+- **Search & Filter**: By device type, frequency, protocol, date range
+- **Statistics Dashboard**: Total captures, unique devices, geographic coverage
+- **Leaderboard**: Top contributors by uploads/verifications
+
+## Wigle.net Analysis & Implementation Strategy
+
+### Key Wigle.net Features to Replicate
+
+#### 1. Submission Workflow
+**Wigle Approach:**
+- CSV upload with standardized format
+- Required fields: MAC, SSID, GPS coords, timestamp
+- Pre-header with client metadata
+- Batch uploads supported
+
+**GigLez Implementation:**
+- Accept .sub/.fff files + GPS JSON/CSV
+- Required fields: GPS (lat/lon), timestamp, signal file
+- Upload via web interface or API
+- Support batch submissions (ZIP of .sub files + manifest.json)
+
+#### 2. Data Storage
+**Wigle Stats (2017):**
+- 349M WiFi networks
+- 7.8M cell towers
+- Billions of observations
+- GPS coordinates for 99%+ of records
+
+**GigLez Architecture:**
+- PostgreSQL with PostGIS for geospatial queries
+- Deduplicate by file hash + GPS proximity
+- Store both raw files and parsed metadata
+- Index by frequency, protocol, location, timestamp
+
+#### 3. Search & Discovery
+**Wigle Features:**
+- Text search (SSID, MAC)
+- Geographic search (bounding box, radius)
+- Advanced filters (encryption, date range)
+- Export to CSV/KML
+
+**GigLez Equivalent:**
+- Search by device type, manufacturer, protocol
+- Geographic search (radius, bounding box)
+- Filter by frequency, modulation, confidence score
+- Export to .sub, JSON, CSV, GeoJSON
+
+#### 4. Mapping Interface
+**Wigle UI:**
+- Zoom-based detail levels
+- Color coding by signal type/quality
+- Click for network details
+- Overlays from entire database
+
+**GigLez UI:**
+- Leaflet.js/Mapbox for mapping
+- Color by device type or frequency band
+- Marker clustering for performance
+- Click for device details (.sub file viewer)
+- Heatmap overlay for density
+
+#### 5. User Accounts & Gamification
+**Wigle System:**
+- User registration required
+- Upload tracking and statistics
+- Leaderboard (global/monthly)
+- Contribution milestones
+
+**GigLez System:**
+- Optional accounts (allow anonymous uploads)
+- Track uploads, identifications, verifications
+- Reputation score for accurate IDs
+- Badges for contributions (first capture in city, 100 devices, etc.)
+
+#### 6. API Access
+**Wigle API:**
+- JSON-based RPC (not REST)
+- Authentication via API token
+- Query endpoints for searching
+- Upload endpoints for submissions
+- Rate limiting
+
+**GigLez API:**
+- RESTful JSON API
+- JWT tokens + API keys
+- Endpoints:
+  - `POST /api/submit` - Upload captures
+  - `GET /api/search` - Query database
+  - `GET /api/devices/{id}` - Device details
+  - `GET /api/heatmap` - Density data
+  - `GET /api/stats` - Platform statistics
+
+### Differences from Wigle
+
+| Feature | Wigle.net | GigLez |
+|---------|-----------|--------|
+| **Data Type** | WiFi, Bluetooth, Cellular | Sub-GHz IoT RF (300-928 MHz) |
+| **Submission Format** | CSV | .sub/.fff files + GPS |
+| **Identification** | MAC/SSID (exact) | Protocol signature matching (fuzzy) |
+| **Focus** | Network mapping | Device type identification |
+| **Privacy** | Public by default | Opt-in sharing, anonymization |
+
+## Technical Specifications
+
+### Supported File Formats
+
+#### 1. Flipper Zero .sub Format
+```
+Filetype: Flipper SubGhz Key File
+Version: 1
+Frequency: 433920000
+Preset: FuriHalSubGhzPresetOok270Async
+Protocol: Princeton
+Bit: 24
+Key: 00 00 00 00 00 95 D5 D4
+TE: 400
+```
+
+**Key Fields for Matching:**
+- `Frequency`: Exact frequency in Hz
+- `Preset`: Modulation type (OOK/FSK)
+- `Protocol`: Protocol name (if decoded)
+- `Bit`: Bit length
+- `Key`: Data payload (hex)
+- `TE`: Timing element (μs)
+
+#### 2. Flipper RAW Format (.sub)
+```
+Filetype: Flipper SubGhz RAW File
+Version: 1
+Frequency: 433920000
+Preset: FuriHalSubGhzPresetOok650Async
+Protocol: RAW
+RAW_Data: 2980 -240 520 -980 520 -980 ...
+```
+
+**Parsing Strategy:**
+- Extract timing patterns
+- Identify repeating sequences
+- Match against known protocols by timing signatures
+- Calculate pulse width statistics
+
+#### 3. .fff Format (Future Feature)
+Support for other firmware formats as needed.
+
+### Submission API Format
+
+#### JSON Manifest
+```json
+{
+  "submission": {
+    "timestamp": "2025-01-11T20:30:00Z",
+    "latitude": 40.7128,
+    "longitude": -74.0060,
+    "altitude": 10.5,
+    "accuracy": 5.0,
+    "session_id": "optional_session_identifier",
+    "user_id": "optional_user_identifier",
+    "device_name": "Flipper Zero",
+    "notes": "Captured near downtown"
+  },
+  "files": [
+    {
+      "filename": "capture_001.sub",
+      "sha256": "abc123...",
+      "size_bytes": 256
+    },
+    {
+      "filename": "capture_002.sub",
+      "sha256": "def456...",
+      "size_bytes": 312
+    }
+  ]
+}
+```
+
+#### CSV Submission Format (Alternative)
+```csv
+latitude,longitude,timestamp,filename,accuracy,altitude
+40.7128,-74.0060,2025-01-11T20:30:00Z,capture_001.sub,5.0,10.5
+40.7129,-74.0061,2025-01-11T20:30:15Z,capture_002.sub,5.0,10.5
+```
+
+### Device Signature Matching Pipeline
+
+#### Step 1: Parse .sub File
+```python
+def parse_sub_file(file_path):
+    """Extract metadata from .sub file"""
+    metadata = {
+        'frequency': None,
+        'protocol': None,
+        'modulation': None,
+        'bit_length': None,
+        'key_data': None,
+        'timing': None,
+        'raw_data': None
+    }
+
+    with open(file_path) as f:
+        for line in f:
+            if ':' in line:
+                key, value = line.split(':', 1)
+                # Map to metadata fields
+
+    return metadata
+```
+
+#### Step 2: Match Against Signature Database
+```python
+def match_signature(metadata):
+    """
+    Match parsed metadata against known signatures
+    Returns: [(device_id, confidence), ...]
+    """
+    matches = []
+
+    # Exact match: protocol + frequency + bit length
+    exact = query_exact_match(
+        metadata['protocol'],
+        metadata['frequency'],
+        metadata['bit_length']
+    )
+    if exact:
+        matches.append((exact.device_id, 1.0))
+
+    # Partial match: protocol + frequency
+    partial = query_partial_match(
+        metadata['protocol'],
+        metadata['frequency']
+    )
+    for match in partial:
+        matches.append((match.device_id, 0.8))
+
+    # Pattern match: bit pattern similarity
+    if metadata['key_data']:
+        pattern_matches = match_bit_patterns(metadata['key_data'])
+        matches.extend(pattern_matches)
+
+    # Timing match: for RAW files
+    if metadata['raw_data']:
+        timing_matches = match_timing_patterns(metadata['raw_data'])
+        matches.extend(timing_matches)
+
+    # Sort by confidence, deduplicate
+    return sorted(set(matches), key=lambda x: x[1], reverse=True)
+```
+
+#### Step 3: Store Results
+```python
+def store_capture(file_path, gps_coords, matches):
+    """Store capture with matched device(s)"""
+    capture = {
+        'latitude': gps_coords['lat'],
+        'longitude': gps_coords['lon'],
+        'timestamp': gps_coords['timestamp'],
+        'file_hash': sha256(file_path),
+        'file_path': upload_to_storage(file_path),
+        **parse_sub_file(file_path)
+    }
+
+    # Store capture
+    capture_id = db.captures.insert(capture)
+
+    # Store top 3 matches
+    for device_id, confidence in matches[:3]:
+        db.capture_matches.insert({
+            'capture_id': capture_id,
+            'device_id': device_id,
+            'confidence': confidence,
+            'method': 'auto'
+        })
+
+    return capture_id
+```
+
+## System Architecture
+
+### Platform Components
+
+```
+┌─────────────────────────────────────────────────────────┐
+│                    Web Interface                         │
+│  - Upload Form (drag .sub files + GPS)                  │
+│  - Interactive Map (Leaflet.js)                          │
+│  - Search & Filter UI                                    │
+│  - Device Database Browser                               │
+└─────────────────────────────────────────────────────────┘
+                           │
+                           ↓
+┌─────────────────────────────────────────────────────────┐
+│                     API Layer (FastAPI)                  │
+│  POST /api/submit    - Upload captures                  │
+│  GET  /api/search    - Query database                   │
+│  GET  /api/devices   - Device catalog                   │
+│  GET  /api/heatmap   - Geographic density               │
+└─────────────────────────────────────────────────────────┘
+                           │
+                           ↓
+┌─────────────────────────────────────────────────────────┐
+│              Processing Pipeline                         │
+│  1. File Parser (.sub/.fff → metadata)                  │
+│  2. Signature Matcher (metadata → device IDs)           │
+│  3. Deduplicator (file hash + GPS proximity)            │
+│  4. Storage Manager (DB + file storage)                 │
+└─────────────────────────────────────────────────────────┘
+                           │
+                           ↓
+┌─────────────────────────────────────────────────────────┐
+│          Database (PostgreSQL + PostGIS)                 │
+│  - captures (GPS + metadata + file refs)                │
+│  - devices (known device types)                          │
+│  - signatures (Flipper/RTL_433 patterns)                │
+│  - users (optional accounts)                             │
+│  - identifications (community verifications)             │
+└─────────────────────────────────────────────────────────┘
+                           │
+                           ↓
+┌─────────────────────────────────────────────────────────┐
+│           Signature Databases (Read-Only)                │
+│  - Flipper Zero .sub collection (1000+ files)           │
+│  - RTL_433 protocol definitions (200+ protocols)        │
+│  - Community-contributed signatures                      │
+└─────────────────────────────────────────────────────────┘
+```
+
+### Key Features
+
+#### Core Platform Features
+- **File Upload**: Drag-and-drop .sub files with GPS coordinates
+- **Automatic Parsing**: Extract frequency, protocol, modulation, data from files
+- **Device Matching**: Identify devices using signature databases
+- **Deduplication**: Prevent duplicate submissions via file hashing
+- **Geospatial Search**: Find captures near location or within bounding box
+- **Heatmap Generation**: Visualize device density by geographic area
+- **Export Data**: Download captures as .sub, JSON, CSV, GeoJSON
+
+#### Community Features
+- **Manual Identification**: Users can add/correct device IDs
+- **Photo Uploads**: Visual evidence of physical devices
+- **Voting System**: Upvote/downvote identifications
+- **Verification**: High-confidence IDs become verified
+- **Contribution Tracking**: Statistics per user
+- **Leaderboard**: Top uploaders and verifiers
+
+#### Privacy Features
+- **Anonymous Uploads**: No account required
+- **GPS Precision Control**: Round coordinates to configurable precision
+- **Private Captures**: Opt-out of public database
+- **BSSID-style Removal**: Allow device signature removal requests
+
+## Development Phases
+
+### Phase 1: Foundation (Weeks 1-2)
+- [x] Database schema design
+- [ ] .sub file parser implementation
+- [ ] GPS coordinate validation
+- [ ] Basic file upload endpoint
+- [ ] Storage backend (local/S3)
+
+### Phase 2: Signature Matching (Weeks 3-4)
+- [ ] Import Flipper Zero .sub database
+- [ ] Import RTL_433 protocol definitions
+- [ ] Build matching engine (exact/partial/pattern)
+- [ ] Confidence scoring algorithm
+- [ ] Match result storage
+
+### Phase 3: Web Interface (Weeks 5-6)
+- [ ] Upload form with drag-and-drop
+- [ ] Map visualization (Leaflet.js)
+- [ ] Search and filter UI
+- [ ] Device detail pages
+- [ ] Statistics dashboard
+
+### Phase 4: API & Integration (Weeks 7-8)
+- [ ] RESTful API endpoints
+- [ ] Authentication (JWT/API keys)
+- [ ] Rate limiting
+- [ ] OpenAPI documentation
+- [ ] Client libraries (Python, JS)
+
+### Phase 5: Community Features (Weeks 9-10)
+- [ ] User accounts (optional)
+- [ ] Manual device identification
+- [ ] Photo upload and display
+- [ ] Voting system
+- [ ] Verification workflow
+
+### Phase 6: Optimization (Weeks 11-12)
+- [ ] Database indexing and optimization
+- [ ] Caching layer (Redis)
+- [ ] CDN for file storage
+- [ ] Batch processing queue
+- [ ] Materialized view updates
+
+## Success Metrics
+
+### Platform Growth
+- Number of unique captures submitted
+- Geographic coverage (cities/countries)
+- Total .sub files processed
+- Database size (captures, devices)
+
+### Community Engagement
+- Active users (uploaders + verifiers)
+- Manual identifications submitted
+- Verification votes cast
+- Photo evidence uploads
+
+### Data Quality
+- Device identification accuracy (verified/total)
+- Average confidence score
+- Duplicate detection rate
+- Geographic precision distribution
+
+### Technical Performance
+- Upload processing time (median)
+- Search query latency (p95)
+- Map render performance
+- API response times
+
+## Technical Constraints
+
+### File Processing
+- .sub file size limits (1MB max recommended)
+- Batch upload limits (100 files or 50MB per request)
+- Supported file formats (.sub initially, .fff future)
+- File parsing timeout (5 seconds per file)
+
+### Geographic Data
+- GPS coordinate precision (6-8 decimal places)
+- Coordinate validation (valid lat/lon ranges)
+- Altitude optional (meters above sea level)
+- Accuracy metadata (horizontal accuracy in meters)
+
+### Database Scalability
+- PostgreSQL with PostGIS for geospatial
+- Partitioning by date for large datasets
+- Index strategy for common queries
+- Materialized views for statistics
+
+### Privacy & Compliance
+- GDPR-style data removal
+- Optional account system
+- GPS anonymization (configurable rounding)
+- No PII in .sub file metadata