giglez/CLAUDE.md

GigLez - IoT RF Device Mapping Platform

Primary Directive

Build a Wigle-like crowdsourced platform for mapping Sub-GHz RF IoT devices. Accept .sub/.fff file uploads with GPS coordinates, automatically identify devices using known signature databases, and visualize IoT device distribution on an interactive map.

Core Objectives

1. Platform-Agnostic RF Signature Submission

• Primary Feature: Accept RF signal captures (.sub, .fff files) with GPS coordinates from ANY capture device
• Submission Requirements:
  • GPS coordinates (latitude/longitude) - REQUIRED
  • Timestamp - REQUIRED
  • .sub or .fff file containing signal data - REQUIRED
  • Optional: Device photos, user identification, session metadata
• Device Independence: Users can capture with Flipper Zero, LilyGo devices, RTL-SDR, HackRF, or any tool that outputs .sub/.fff format

2. Automatic Device Identification

Extract and identify devices from raw RF captures by:

• File Parsing: Extract protocol, frequency, modulation, bit patterns from .sub/.fff files
• Database Matching: Compare against known signature databases (Flipper Zero, RTL_433)
• Confidence Scoring: Rank matches by similarity (exact, partial, pattern-based)
• Community Verification: Allow users to confirm/correct automatic identifications

3. Wigle-Style Mapping Platform

Provide web-based visualization similar to Wigle.net:

• Interactive Map: Display captured devices with GPS markers
• Heatmap View: Show device density by geographic area
• Search & Filter: By device type, frequency, protocol, date range
• Statistics Dashboard: Total captures, unique devices, geographic coverage
• Leaderboard: Top contributors by uploads/verifications

Wigle.net Analysis & Implementation Strategy

Key Wigle.net Features to Replicate

1. Submission Workflow

Wigle Approach:

• CSV upload with standardized format
• Required fields: MAC, SSID, GPS coords, timestamp
• Pre-header with client metadata
• Batch uploads supported

GigLez Implementation:

• Accept .sub/.fff files + GPS JSON/CSV
• Required fields: GPS (lat/lon), timestamp, signal file
• Upload via web interface or API
• Support batch submissions (ZIP of .sub files + manifest.json)

2. Data Storage

Wigle Stats (2017):

• 349M WiFi networks
• 7.8M cell towers
• Billions of observations
• GPS coordinates for 99%+ of records

GigLez Architecture:

• PostgreSQL with PostGIS for geospatial queries
• Deduplicate by file hash + GPS proximity
• Store both raw files and parsed metadata
• Index by frequency, protocol, location, timestamp

3. Search & Discovery

Wigle Features:

• Text search (SSID, MAC)
• Geographic search (bounding box, radius)
• Advanced filters (encryption, date range)
• Export to CSV/KML

GigLez Equivalent:

• Search by device type, manufacturer, protocol
• Geographic search (radius, bounding box)
• Filter by frequency, modulation, confidence score
• Export to .sub, JSON, CSV, GeoJSON

4. Mapping Interface

Wigle UI:

• Zoom-based detail levels
• Color coding by signal type/quality
• Click for network details
• Overlays from entire database

GigLez UI:

• Leaflet.js/Mapbox for mapping
• Color by device type or frequency band
• Marker clustering for performance
• Click for device details (.sub file viewer)
• Heatmap overlay for density

5. User Accounts & Gamification

Wigle System:

• User registration required
• Upload tracking and statistics
• Leaderboard (global/monthly)
• Contribution milestones

GigLez System:

• Optional accounts (allow anonymous uploads)
• Track uploads, identifications, verifications
• Reputation score for accurate IDs
• Badges for contributions (first capture in city, 100 devices, etc.)

6. API Access

Wigle API:

• JSON-based RPC (not REST)
• Authentication via API token
• Query endpoints for searching
• Upload endpoints for submissions
• Rate limiting

GigLez API:

• RESTful JSON API
• JWT tokens + API keys
• Endpoints:
  • POST /api/submit - Upload captures
  • GET /api/search - Query database
  • GET /api/devices/{id} - Device details
  • GET /api/heatmap - Density data
  • GET /api/stats - Platform statistics

Differences from Wigle

Feature	Wigle.net	GigLez
Data Type	WiFi, Bluetooth, Cellular	Sub-GHz IoT RF (300-928 MHz)
Submission Format	CSV	.sub/.fff files + GPS
Identification	MAC/SSID (exact)	Protocol signature matching (fuzzy)
Focus	Network mapping	Device type identification
Privacy	Public by default	Opt-in sharing, anonymization

Technical Specifications

Supported File Formats

1. Flipper Zero .sub Format

Filetype: Flipper SubGhz Key File
Version: 1
Frequency: 433920000
Preset: FuriHalSubGhzPresetOok270Async
Protocol: Princeton
Bit: 24
Key: 00 00 00 00 00 95 D5 D4
TE: 400

Key Fields for Matching:

• Frequency: Exact frequency in Hz
• Preset: Modulation type (OOK/FSK)
• Protocol: Protocol name (if decoded)
• Bit: Bit length
• Key: Data payload (hex)
• TE: Timing element (μs)

2. Flipper RAW Format (.sub)

Filetype: Flipper SubGhz RAW File
Version: 1
Frequency: 433920000
Preset: FuriHalSubGhzPresetOok650Async
Protocol: RAW
RAW_Data: 2980 -240 520 -980 520 -980 ...

Parsing Strategy:

• Extract timing patterns
• Identify repeating sequences
• Match against known protocols by timing signatures
• Calculate pulse width statistics

3. .fff Format (Future Feature)

Support for other firmware formats as needed.

Submission API Format

JSON Manifest

{
  "submission": {
    "timestamp": "2025-01-11T20:30:00Z",
    "latitude": 40.7128,
    "longitude": -74.0060,
    "altitude": 10.5,
    "accuracy": 5.0,
    "session_id": "optional_session_identifier",
    "user_id": "optional_user_identifier",
    "device_name": "Flipper Zero",
    "notes": "Captured near downtown"
  },
  "files": [
    {
      "filename": "capture_001.sub",
      "sha256": "abc123...",
      "size_bytes": 256
    },
    {
      "filename": "capture_002.sub",
      "sha256": "def456...",
      "size_bytes": 312
    }
  ]
}

CSV Submission Format (Alternative)

latitude,longitude,timestamp,filename,accuracy,altitude
40.7128,-74.0060,2025-01-11T20:30:00Z,capture_001.sub,5.0,10.5
40.7129,-74.0061,2025-01-11T20:30:15Z,capture_002.sub,5.0,10.5

Device Signature Matching Pipeline

Step 1: Parse .sub File

def parse_sub_file(file_path):
    """Extract metadata from .sub file"""
    metadata = {
        'frequency': None,
        'protocol': None,
        'modulation': None,
        'bit_length': None,
        'key_data': None,
        'timing': None,
        'raw_data': None
    }

    with open(file_path) as f:
        for line in f:
            if ':' in line:
                key, value = line.split(':', 1)
                # Map to metadata fields

    return metadata

Step 2: Match Against Signature Database

def match_signature(metadata):
    """
    Match parsed metadata against known signatures
    Returns: [(device_id, confidence), ...]
    """
    matches = []

    # Exact match: protocol + frequency + bit length
    exact = query_exact_match(
        metadata['protocol'],
        metadata['frequency'],
        metadata['bit_length']
    )
    if exact:
        matches.append((exact.device_id, 1.0))

    # Partial match: protocol + frequency
    partial = query_partial_match(
        metadata['protocol'],
        metadata['frequency']
    )
    for match in partial:
        matches.append((match.device_id, 0.8))

    # Pattern match: bit pattern similarity
    if metadata['key_data']:
        pattern_matches = match_bit_patterns(metadata['key_data'])
        matches.extend(pattern_matches)

    # Timing match: for RAW files
    if metadata['raw_data']:
        timing_matches = match_timing_patterns(metadata['raw_data'])
        matches.extend(timing_matches)

    # Sort by confidence, deduplicate
    return sorted(set(matches), key=lambda x: x[1], reverse=True)

Step 3: Store Results

def store_capture(file_path, gps_coords, matches):
    """Store capture with matched device(s)"""
    capture = {
        'latitude': gps_coords['lat'],
        'longitude': gps_coords['lon'],
        'timestamp': gps_coords['timestamp'],
        'file_hash': sha256(file_path),
        'file_path': upload_to_storage(file_path),
        **parse_sub_file(file_path)
    }

    # Store capture
    capture_id = db.captures.insert(capture)

    # Store top 3 matches
    for device_id, confidence in matches[:3]:
        db.capture_matches.insert({
            'capture_id': capture_id,
            'device_id': device_id,
            'confidence': confidence,
            'method': 'auto'
        })

    return capture_id

System Architecture

Platform Components

┌─────────────────────────────────────────────────────────┐
│                    Web Interface                         │
│  - Upload Form (drag .sub files + GPS)                  │
│  - Interactive Map (Leaflet.js)                          │
│  - Search & Filter UI                                    │
│  - Device Database Browser                               │
└─────────────────────────────────────────────────────────┘
                           │
                           ↓
┌─────────────────────────────────────────────────────────┐
│                     API Layer (FastAPI)                  │
│  POST /api/submit    - Upload captures                  │
│  GET  /api/search    - Query database                   │
│  GET  /api/devices   - Device catalog                   │
│  GET  /api/heatmap   - Geographic density               │
└─────────────────────────────────────────────────────────┘
                           │
                           ↓
┌─────────────────────────────────────────────────────────┐
│              Processing Pipeline                         │
│  1. File Parser (.sub/.fff → metadata)                  │
│  2. Signature Matcher (metadata → device IDs)           │
│  3. Deduplicator (file hash + GPS proximity)            │
│  4. Storage Manager (DB + file storage)                 │
└─────────────────────────────────────────────────────────┘
                           │
                           ↓
┌─────────────────────────────────────────────────────────┐
│          Database (PostgreSQL + PostGIS)                 │
│  - captures (GPS + metadata + file refs)                │
│  - devices (known device types)                          │
│  - signatures (Flipper/RTL_433 patterns)                │
│  - users (optional accounts)                             │
│  - identifications (community verifications)             │
└─────────────────────────────────────────────────────────┘
                           │
                           ↓
┌─────────────────────────────────────────────────────────┐
│           Signature Databases (Read-Only)                │
│  - Flipper Zero .sub collection (1000+ files)           │
│  - RTL_433 protocol definitions (200+ protocols)        │
│  - Community-contributed signatures                      │
└─────────────────────────────────────────────────────────┘

Key Features

Core Platform Features

• File Upload: Drag-and-drop .sub files with GPS coordinates
• Automatic Parsing: Extract frequency, protocol, modulation, data from files
• Device Matching: Identify devices using signature databases
• Deduplication: Prevent duplicate submissions via file hashing
• Geospatial Search: Find captures near location or within bounding box
• Heatmap Generation: Visualize device density by geographic area
• Export Data: Download captures as .sub, JSON, CSV, GeoJSON

Community Features

• Manual Identification: Users can add/correct device IDs
• Photo Uploads: Visual evidence of physical devices
• Voting System: Upvote/downvote identifications
• Verification: High-confidence IDs become verified
• Contribution Tracking: Statistics per user
• Leaderboard: Top uploaders and verifiers

Privacy Features

• Anonymous Uploads: No account required
• GPS Precision Control: Round coordinates to configurable precision
• Private Captures: Opt-out of public database
• BSSID-style Removal: Allow device signature removal requests

Development Phases

Phase 1: Foundation (Weeks 1-2)

• Database schema design
• .sub file parser implementation
• GPS coordinate validation
• Basic file upload endpoint
• Storage backend (local/S3)

Phase 2: Signature Matching (Weeks 3-4)

• Import Flipper Zero .sub database
• Import RTL_433 protocol definitions
• Build matching engine (exact/partial/pattern)
• Confidence scoring algorithm
• Match result storage

Phase 3: Web Interface (Weeks 5-6)

• Upload form with drag-and-drop
• Map visualization (Leaflet.js)
• Search and filter UI
• Device detail pages
• Statistics dashboard

Phase 4: API & Integration (Weeks 7-8)

• RESTful API endpoints
• Authentication (JWT/API keys)
• Rate limiting
• OpenAPI documentation
• Client libraries (Python, JS)

Phase 5: Community Features (Weeks 9-10)

• User accounts (optional)
• Manual device identification
• Photo upload and display
• Voting system
• Verification workflow

Phase 6: Optimization (Weeks 11-12)

• Database indexing and optimization
• Caching layer (Redis)
• CDN for file storage
• Batch processing queue
• Materialized view updates

Success Metrics

Platform Growth

• Number of unique captures submitted
• Geographic coverage (cities/countries)
• Total .sub files processed
• Database size (captures, devices)

Community Engagement

• Active users (uploaders + verifiers)
• Manual identifications submitted
• Verification votes cast
• Photo evidence uploads

Data Quality

• Device identification accuracy (verified/total)
• Average confidence score
• Duplicate detection rate
• Geographic precision distribution

Technical Performance

• Upload processing time (median)
• Search query latency (p95)
• Map render performance
• API response times

Technical Constraints

File Processing

• .sub file size limits (1MB max recommended)
• Batch upload limits (100 files or 50MB per request)
• Supported file formats (.sub initially, .fff future)
• File parsing timeout (5 seconds per file)

Geographic Data

• GPS coordinate precision (6-8 decimal places)
• Coordinate validation (valid lat/lon ranges)
• Altitude optional (meters above sea level)
• Accuracy metadata (horizontal accuracy in meters)

Database Scalability

• PostgreSQL with PostGIS for geospatial
• Partitioning by date for large datasets
• Index strategy for common queries
• Materialized views for statistics

Privacy & Compliance

• GDPR-style data removal
• Optional account system
• GPS anonymization (configurable rounding)
• No PII in .sub file metadata

Current Implementation Status (Iteration 6/6 Complete)

Device Identification Architecture

Final 6-Component Scoring System:

1. Timing Analysis (40% weight) - src/matcher/timing_analyzer.py

   • Dual timing validation (SHORT + LONG pulses)
   • Weighted average: SHORT (60%) + LONG (40%)
   • Multi-method extraction (K-means, histogram, percentile)
   • IQR outlier removal (2.5x threshold)
   • Noise tolerance: 15% jitter tested successfully

2. Preamble Detection (25% weight) - src/matcher/preamble_detector.py

   • 4 detection methods: long_burst, alternating, sync_word, custom
   • Sorted pattern matching (longest first)
   • Preamble boost: +5% for >90% preamble match + >80% overall score
   • Highly discriminative for protocol families

3. Timing Ratio Matching (20% weight) - src/matcher/pattern_decoder.py

   • Compare LONG/SHORT pulse ratios
   • Highly discriminative (2:1 vs 3:1 separates families)
   • Catches relative timing errors

4. Frequency Fingerprinting (10% weight) - src/matcher/frequency_fingerprint.py

   • ISM band classification (315/433/868/915 MHz)
   • Tighter tolerance: ±100 kHz (strict), gradual falloff to ±500 kHz
   • Reduces search space from 299 → ~20-30 candidates

5. Bit Count Matching (5% weight) - src/matcher/pattern_decoder.py

   • Relaxed scoring (unreliable in synthetic signals)
   • Flexible range validation
   • Reduced from 20% weight in iteration 5

6. Uniqueness Bonus (0-20% boost) - src/matcher/pattern_decoder.py

   • +20% bonus for unique timing (only 1 similar protocol)
   • +15% for 2 similar protocols, +10% for 3, +5% for 4-5
   • Discriminates rare vs. common timing signatures

Current Accuracy Metrics (Benchmark Results - Iteration 6)

Test Suite: 12 synthetic signals across 10 protocols

Overall Performance:

• Top-1 Accuracy: 33.3% (4/12 correct)
• Top-3 Accuracy: 50.0% (6/12 in top 3) ⬆️ +17% improvement
• Top-5 Accuracy: 50.0%
• Target: ≥25% ✅ PASSED

Confidence Distribution:

• High (>80%): 66.7% (tighter scoring reduces overconfidence)
• Medium (50-80%): 25.0%
• Low (<50%): 8.3%

Processing Performance:

• Avg Parse Time: 0.26 ms
• Avg Match Time: 132.81 ms
• Total: 133.07 ms per signal (15% faster than iteration 5)

Key Findings (Iteration 6):

• ✅ Noise Tolerance: Successfully handles 15% jitter
• ✅ Preamble Detection: Critical for discrimination (alternating patterns excel)
• ✅ Timing Ratio: LONG/SHORT ratio highly discriminative (2:1 vs 3:1)
• ✅ Family Matching: Acurite 609TXC → Acurite 896 (correct family, rank 2)
• ✅ Test Data Fixed: Corrected Acurite and Oregon timing parameters
• ⚠️ Princeton: Still failing (SimpliSafe mismatch) - needs protocol tuning
• ⚠️ Generic Protocols: Need more database entries (Nexus, Honeywell, Generic Doorbell not in DB)

Next Steps for Further Improvement

1. Add Missing Protocols: Nexus Temperature-Humidity, Honeywell Security, Generic Doorbell to database
2. Princeton Tuning: Investigate SimpliSafe mismatch - may need protocol signature refinement
3. TPMS Family Logic: Toyota vs Schrader - add manufacturer-specific heuristics
4. Real-World Testing: Benchmark against actual Flipper Zero captures (not synthetic)
5. Protocol Family Scoring: Bonus for partial name matches (Acurite 896 vs Acurite 609TXC)

Last Updated: 2026-02-15 - Iteration 5/5 complete