475 lines
16 KiB
Markdown
475 lines
16 KiB
Markdown
# GigLez - IoT RF Device Mapping Platform
|
|
|
|
## Primary Directive
|
|
|
|
Build a Wigle-like crowdsourced platform for mapping Sub-GHz RF IoT devices. Accept .sub/.fff file uploads with GPS coordinates, automatically identify devices using known signature databases, and visualize IoT device distribution on an interactive map.
|
|
|
|
## Core Objectives
|
|
|
|
### 1. Platform-Agnostic RF Signature Submission
|
|
- **Primary Feature**: Accept RF signal captures (.sub, .fff files) with GPS coordinates from ANY capture device
|
|
- **Submission Requirements**:
|
|
- GPS coordinates (latitude/longitude) - REQUIRED
|
|
- Timestamp - REQUIRED
|
|
- .sub or .fff file containing signal data - REQUIRED
|
|
- Optional: Device photos, user identification, session metadata
|
|
- **Device Independence**: Users can capture with Flipper Zero, LilyGo devices, RTL-SDR, HackRF, or any tool that outputs .sub/.fff format
|
|
|
|
### 2. Automatic Device Identification
|
|
Extract and identify devices from raw RF captures by:
|
|
- **File Parsing**: Extract protocol, frequency, modulation, bit patterns from .sub/.fff files
|
|
- **Database Matching**: Compare against known signature databases (Flipper Zero, RTL_433)
|
|
- **Confidence Scoring**: Rank matches by similarity (exact, partial, pattern-based)
|
|
- **Community Verification**: Allow users to confirm/correct automatic identifications
|
|
|
|
### 3. Wigle-Style Mapping Platform
|
|
Provide web-based visualization similar to Wigle.net:
|
|
- **Interactive Map**: Display captured devices with GPS markers
|
|
- **Heatmap View**: Show device density by geographic area
|
|
- **Search & Filter**: By device type, frequency, protocol, date range
|
|
- **Statistics Dashboard**: Total captures, unique devices, geographic coverage
|
|
- **Leaderboard**: Top contributors by uploads/verifications
|
|
|
|
## Wigle.net Analysis & Implementation Strategy
|
|
|
|
### Key Wigle.net Features to Replicate
|
|
|
|
#### 1. Submission Workflow
|
|
**Wigle Approach:**
|
|
- CSV upload with standardized format
|
|
- Required fields: MAC, SSID, GPS coords, timestamp
|
|
- Pre-header with client metadata
|
|
- Batch uploads supported
|
|
|
|
**GigLez Implementation:**
|
|
- Accept .sub/.fff files + GPS JSON/CSV
|
|
- Required fields: GPS (lat/lon), timestamp, signal file
|
|
- Upload via web interface or API
|
|
- Support batch submissions (ZIP of .sub files + manifest.json)
|
|
|
|
#### 2. Data Storage
|
|
**Wigle Stats (2017):**
|
|
- 349M WiFi networks
|
|
- 7.8M cell towers
|
|
- Billions of observations
|
|
- GPS coordinates for 99%+ of records
|
|
|
|
**GigLez Architecture:**
|
|
- PostgreSQL with PostGIS for geospatial queries
|
|
- Deduplicate by file hash + GPS proximity
|
|
- Store both raw files and parsed metadata
|
|
- Index by frequency, protocol, location, timestamp
|
|
|
|
#### 3. Search & Discovery
|
|
**Wigle Features:**
|
|
- Text search (SSID, MAC)
|
|
- Geographic search (bounding box, radius)
|
|
- Advanced filters (encryption, date range)
|
|
- Export to CSV/KML
|
|
|
|
**GigLez Equivalent:**
|
|
- Search by device type, manufacturer, protocol
|
|
- Geographic search (radius, bounding box)
|
|
- Filter by frequency, modulation, confidence score
|
|
- Export to .sub, JSON, CSV, GeoJSON
|
|
|
|
#### 4. Mapping Interface
|
|
**Wigle UI:**
|
|
- Zoom-based detail levels
|
|
- Color coding by signal type/quality
|
|
- Click for network details
|
|
- Overlays from entire database
|
|
|
|
**GigLez UI:**
|
|
- Leaflet.js/Mapbox for mapping
|
|
- Color by device type or frequency band
|
|
- Marker clustering for performance
|
|
- Click for device details (.sub file viewer)
|
|
- Heatmap overlay for density
|
|
|
|
#### 5. User Accounts & Gamification
|
|
**Wigle System:**
|
|
- User registration required
|
|
- Upload tracking and statistics
|
|
- Leaderboard (global/monthly)
|
|
- Contribution milestones
|
|
|
|
**GigLez System:**
|
|
- Optional accounts (allow anonymous uploads)
|
|
- Track uploads, identifications, verifications
|
|
- Reputation score for accurate IDs
|
|
- Badges for contributions (first capture in city, 100 devices, etc.)
|
|
|
|
#### 6. API Access
|
|
**Wigle API:**
|
|
- JSON-based RPC (not REST)
|
|
- Authentication via API token
|
|
- Query endpoints for searching
|
|
- Upload endpoints for submissions
|
|
- Rate limiting
|
|
|
|
**GigLez API:**
|
|
- RESTful JSON API
|
|
- JWT tokens + API keys
|
|
- Endpoints:
|
|
- `POST /api/submit` - Upload captures
|
|
- `GET /api/search` - Query database
|
|
- `GET /api/devices/{id}` - Device details
|
|
- `GET /api/heatmap` - Density data
|
|
- `GET /api/stats` - Platform statistics
|
|
|
|
### Differences from Wigle
|
|
|
|
| Feature | Wigle.net | GigLez |
|
|
|---------|-----------|--------|
|
|
| **Data Type** | WiFi, Bluetooth, Cellular | Sub-GHz IoT RF (300-928 MHz) |
|
|
| **Submission Format** | CSV | .sub/.fff files + GPS |
|
|
| **Identification** | MAC/SSID (exact) | Protocol signature matching (fuzzy) |
|
|
| **Focus** | Network mapping | Device type identification |
|
|
| **Privacy** | Public by default | Opt-in sharing, anonymization |
|
|
|
|
## Technical Specifications
|
|
|
|
### Supported File Formats
|
|
|
|
#### 1. Flipper Zero .sub Format
|
|
```
|
|
Filetype: Flipper SubGhz Key File
|
|
Version: 1
|
|
Frequency: 433920000
|
|
Preset: FuriHalSubGhzPresetOok270Async
|
|
Protocol: Princeton
|
|
Bit: 24
|
|
Key: 00 00 00 00 00 95 D5 D4
|
|
TE: 400
|
|
```
|
|
|
|
**Key Fields for Matching:**
|
|
- `Frequency`: Exact frequency in Hz
|
|
- `Preset`: Modulation type (OOK/FSK)
|
|
- `Protocol`: Protocol name (if decoded)
|
|
- `Bit`: Bit length
|
|
- `Key`: Data payload (hex)
|
|
- `TE`: Timing element (μs)
|
|
|
|
#### 2. Flipper RAW Format (.sub)
|
|
```
|
|
Filetype: Flipper SubGhz RAW File
|
|
Version: 1
|
|
Frequency: 433920000
|
|
Preset: FuriHalSubGhzPresetOok650Async
|
|
Protocol: RAW
|
|
RAW_Data: 2980 -240 520 -980 520 -980 ...
|
|
```
|
|
|
|
**Parsing Strategy:**
|
|
- Extract timing patterns
|
|
- Identify repeating sequences
|
|
- Match against known protocols by timing signatures
|
|
- Calculate pulse width statistics
|
|
|
|
#### 3. .fff Format (Future Feature)
|
|
Support for other firmware formats as needed.
|
|
|
|
### Submission API Format
|
|
|
|
#### JSON Manifest
|
|
```json
|
|
{
|
|
"submission": {
|
|
"timestamp": "2025-01-11T20:30:00Z",
|
|
"latitude": 40.7128,
|
|
"longitude": -74.0060,
|
|
"altitude": 10.5,
|
|
"accuracy": 5.0,
|
|
"session_id": "optional_session_identifier",
|
|
"user_id": "optional_user_identifier",
|
|
"device_name": "Flipper Zero",
|
|
"notes": "Captured near downtown"
|
|
},
|
|
"files": [
|
|
{
|
|
"filename": "capture_001.sub",
|
|
"sha256": "abc123...",
|
|
"size_bytes": 256
|
|
},
|
|
{
|
|
"filename": "capture_002.sub",
|
|
"sha256": "def456...",
|
|
"size_bytes": 312
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
#### CSV Submission Format (Alternative)
|
|
```csv
|
|
latitude,longitude,timestamp,filename,accuracy,altitude
|
|
40.7128,-74.0060,2025-01-11T20:30:00Z,capture_001.sub,5.0,10.5
|
|
40.7129,-74.0061,2025-01-11T20:30:15Z,capture_002.sub,5.0,10.5
|
|
```
|
|
|
|
### Device Signature Matching Pipeline
|
|
|
|
#### Step 1: Parse .sub File
|
|
```python
|
|
def parse_sub_file(file_path):
|
|
"""Extract metadata from .sub file"""
|
|
metadata = {
|
|
'frequency': None,
|
|
'protocol': None,
|
|
'modulation': None,
|
|
'bit_length': None,
|
|
'key_data': None,
|
|
'timing': None,
|
|
'raw_data': None
|
|
}
|
|
|
|
with open(file_path) as f:
|
|
for line in f:
|
|
if ':' in line:
|
|
key, value = line.split(':', 1)
|
|
# Map to metadata fields
|
|
|
|
return metadata
|
|
```
|
|
|
|
#### Step 2: Match Against Signature Database
|
|
```python
|
|
def match_signature(metadata):
|
|
"""
|
|
Match parsed metadata against known signatures
|
|
Returns: [(device_id, confidence), ...]
|
|
"""
|
|
matches = []
|
|
|
|
# Exact match: protocol + frequency + bit length
|
|
exact = query_exact_match(
|
|
metadata['protocol'],
|
|
metadata['frequency'],
|
|
metadata['bit_length']
|
|
)
|
|
if exact:
|
|
matches.append((exact.device_id, 1.0))
|
|
|
|
# Partial match: protocol + frequency
|
|
partial = query_partial_match(
|
|
metadata['protocol'],
|
|
metadata['frequency']
|
|
)
|
|
for match in partial:
|
|
matches.append((match.device_id, 0.8))
|
|
|
|
# Pattern match: bit pattern similarity
|
|
if metadata['key_data']:
|
|
pattern_matches = match_bit_patterns(metadata['key_data'])
|
|
matches.extend(pattern_matches)
|
|
|
|
# Timing match: for RAW files
|
|
if metadata['raw_data']:
|
|
timing_matches = match_timing_patterns(metadata['raw_data'])
|
|
matches.extend(timing_matches)
|
|
|
|
# Sort by confidence, deduplicate
|
|
return sorted(set(matches), key=lambda x: x[1], reverse=True)
|
|
```
|
|
|
|
#### Step 3: Store Results
|
|
```python
|
|
def store_capture(file_path, gps_coords, matches):
|
|
"""Store capture with matched device(s)"""
|
|
capture = {
|
|
'latitude': gps_coords['lat'],
|
|
'longitude': gps_coords['lon'],
|
|
'timestamp': gps_coords['timestamp'],
|
|
'file_hash': sha256(file_path),
|
|
'file_path': upload_to_storage(file_path),
|
|
**parse_sub_file(file_path)
|
|
}
|
|
|
|
# Store capture
|
|
capture_id = db.captures.insert(capture)
|
|
|
|
# Store top 3 matches
|
|
for device_id, confidence in matches[:3]:
|
|
db.capture_matches.insert({
|
|
'capture_id': capture_id,
|
|
'device_id': device_id,
|
|
'confidence': confidence,
|
|
'method': 'auto'
|
|
})
|
|
|
|
return capture_id
|
|
```
|
|
|
|
## System Architecture
|
|
|
|
### Platform Components
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────┐
|
|
│ Web Interface │
|
|
│ - Upload Form (drag .sub files + GPS) │
|
|
│ - Interactive Map (Leaflet.js) │
|
|
│ - Search & Filter UI │
|
|
│ - Device Database Browser │
|
|
└─────────────────────────────────────────────────────────┘
|
|
│
|
|
↓
|
|
┌─────────────────────────────────────────────────────────┐
|
|
│ API Layer (FastAPI) │
|
|
│ POST /api/submit - Upload captures │
|
|
│ GET /api/search - Query database │
|
|
│ GET /api/devices - Device catalog │
|
|
│ GET /api/heatmap - Geographic density │
|
|
└─────────────────────────────────────────────────────────┘
|
|
│
|
|
↓
|
|
┌─────────────────────────────────────────────────────────┐
|
|
│ Processing Pipeline │
|
|
│ 1. File Parser (.sub/.fff → metadata) │
|
|
│ 2. Signature Matcher (metadata → device IDs) │
|
|
│ 3. Deduplicator (file hash + GPS proximity) │
|
|
│ 4. Storage Manager (DB + file storage) │
|
|
└─────────────────────────────────────────────────────────┘
|
|
│
|
|
↓
|
|
┌─────────────────────────────────────────────────────────┐
|
|
│ Database (PostgreSQL + PostGIS) │
|
|
│ - captures (GPS + metadata + file refs) │
|
|
│ - devices (known device types) │
|
|
│ - signatures (Flipper/RTL_433 patterns) │
|
|
│ - users (optional accounts) │
|
|
│ - identifications (community verifications) │
|
|
└─────────────────────────────────────────────────────────┘
|
|
│
|
|
↓
|
|
┌─────────────────────────────────────────────────────────┐
|
|
│ Signature Databases (Read-Only) │
|
|
│ - Flipper Zero .sub collection (1000+ files) │
|
|
│ - RTL_433 protocol definitions (200+ protocols) │
|
|
│ - Community-contributed signatures │
|
|
└─────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
### Key Features
|
|
|
|
#### Core Platform Features
|
|
- **File Upload**: Drag-and-drop .sub files with GPS coordinates
|
|
- **Automatic Parsing**: Extract frequency, protocol, modulation, data from files
|
|
- **Device Matching**: Identify devices using signature databases
|
|
- **Deduplication**: Prevent duplicate submissions via file hashing
|
|
- **Geospatial Search**: Find captures near location or within bounding box
|
|
- **Heatmap Generation**: Visualize device density by geographic area
|
|
- **Export Data**: Download captures as .sub, JSON, CSV, GeoJSON
|
|
|
|
#### Community Features
|
|
- **Manual Identification**: Users can add/correct device IDs
|
|
- **Photo Uploads**: Visual evidence of physical devices
|
|
- **Voting System**: Upvote/downvote identifications
|
|
- **Verification**: High-confidence IDs become verified
|
|
- **Contribution Tracking**: Statistics per user
|
|
- **Leaderboard**: Top uploaders and verifiers
|
|
|
|
#### Privacy Features
|
|
- **Anonymous Uploads**: No account required
|
|
- **GPS Precision Control**: Round coordinates to configurable precision
|
|
- **Private Captures**: Opt-out of public database
|
|
- **BSSID-style Removal**: Allow device signature removal requests
|
|
|
|
## Development Phases
|
|
|
|
### Phase 1: Foundation (Weeks 1-2)
|
|
- [x] Database schema design
|
|
- [ ] .sub file parser implementation
|
|
- [ ] GPS coordinate validation
|
|
- [ ] Basic file upload endpoint
|
|
- [ ] Storage backend (local/S3)
|
|
|
|
### Phase 2: Signature Matching (Weeks 3-4)
|
|
- [ ] Import Flipper Zero .sub database
|
|
- [ ] Import RTL_433 protocol definitions
|
|
- [ ] Build matching engine (exact/partial/pattern)
|
|
- [ ] Confidence scoring algorithm
|
|
- [ ] Match result storage
|
|
|
|
### Phase 3: Web Interface (Weeks 5-6)
|
|
- [ ] Upload form with drag-and-drop
|
|
- [ ] Map visualization (Leaflet.js)
|
|
- [ ] Search and filter UI
|
|
- [ ] Device detail pages
|
|
- [ ] Statistics dashboard
|
|
|
|
### Phase 4: API & Integration (Weeks 7-8)
|
|
- [ ] RESTful API endpoints
|
|
- [ ] Authentication (JWT/API keys)
|
|
- [ ] Rate limiting
|
|
- [ ] OpenAPI documentation
|
|
- [ ] Client libraries (Python, JS)
|
|
|
|
### Phase 5: Community Features (Weeks 9-10)
|
|
- [ ] User accounts (optional)
|
|
- [ ] Manual device identification
|
|
- [ ] Photo upload and display
|
|
- [ ] Voting system
|
|
- [ ] Verification workflow
|
|
|
|
### Phase 6: Optimization (Weeks 11-12)
|
|
- [ ] Database indexing and optimization
|
|
- [ ] Caching layer (Redis)
|
|
- [ ] CDN for file storage
|
|
- [ ] Batch processing queue
|
|
- [ ] Materialized view updates
|
|
|
|
## Success Metrics
|
|
|
|
### Platform Growth
|
|
- Number of unique captures submitted
|
|
- Geographic coverage (cities/countries)
|
|
- Total .sub files processed
|
|
- Database size (captures, devices)
|
|
|
|
### Community Engagement
|
|
- Active users (uploaders + verifiers)
|
|
- Manual identifications submitted
|
|
- Verification votes cast
|
|
- Photo evidence uploads
|
|
|
|
### Data Quality
|
|
- Device identification accuracy (verified/total)
|
|
- Average confidence score
|
|
- Duplicate detection rate
|
|
- Geographic precision distribution
|
|
|
|
### Technical Performance
|
|
- Upload processing time (median)
|
|
- Search query latency (p95)
|
|
- Map render performance
|
|
- API response times
|
|
|
|
## Technical Constraints
|
|
|
|
### File Processing
|
|
- .sub file size limits (1MB max recommended)
|
|
- Batch upload limits (100 files or 50MB per request)
|
|
- Supported file formats (.sub initially, .fff future)
|
|
- File parsing timeout (5 seconds per file)
|
|
|
|
### Geographic Data
|
|
- GPS coordinate precision (6-8 decimal places)
|
|
- Coordinate validation (valid lat/lon ranges)
|
|
- Altitude optional (meters above sea level)
|
|
- Accuracy metadata (horizontal accuracy in meters)
|
|
|
|
### Database Scalability
|
|
- PostgreSQL with PostGIS for geospatial
|
|
- Partitioning by date for large datasets
|
|
- Index strategy for common queries
|
|
- Materialized views for statistics
|
|
|
|
### Privacy & Compliance
|
|
- GDPR-style data removal
|
|
- Optional account system
|
|
- GPS anonymization (configurable rounding)
|
|
- No PII in .sub file metadata
|