giglez/CLAUDE.md

# GigLez - IoT RF Device Mapping Platform

## Primary Directive

Build a Wigle-like crowdsourced platform for mapping Sub-GHz RF IoT devices. Accept .sub/.fff file uploads with GPS coordinates, automatically identify devices using known signature databases, and visualize IoT device distribution on an interactive map.

## Core Objectives

### 1. Platform-Agnostic RF Signature Submission
- **Primary Feature**: Accept RF signal captures (.sub, .fff files) with GPS coordinates from ANY capture device
- **Submission Requirements**:
  - GPS coordinates (latitude/longitude) - REQUIRED
  - Timestamp - REQUIRED
  - .sub or .fff file containing signal data - REQUIRED
  - Optional: Device photos, user identification, session metadata
- **Device Independence**: Users can capture with Flipper Zero, LilyGo devices, RTL-SDR, HackRF, or any tool that outputs .sub/.fff format

### 2. Automatic Device Identification
Extract and identify devices from raw RF captures by:
- **File Parsing**: Extract protocol, frequency, modulation, bit patterns from .sub/.fff files
- **Database Matching**: Compare against known signature databases (Flipper Zero, RTL_433)
- **Confidence Scoring**: Rank matches by similarity (exact, partial, pattern-based)
- **Community Verification**: Allow users to confirm/correct automatic identifications

### 3. Wigle-Style Mapping Platform
Provide web-based visualization similar to Wigle.net:
- **Interactive Map**: Display captured devices with GPS markers
- **Heatmap View**: Show device density by geographic area
- **Search & Filter**: By device type, frequency, protocol, date range
- **Statistics Dashboard**: Total captures, unique devices, geographic coverage
- **Leaderboard**: Top contributors by uploads/verifications

## Wigle.net Analysis & Implementation Strategy

### Key Wigle.net Features to Replicate

#### 1. Submission Workflow
**Wigle Approach:**
- CSV upload with standardized format
- Required fields: MAC, SSID, GPS coords, timestamp
- Pre-header with client metadata
- Batch uploads supported

**GigLez Implementation:**
- Accept .sub/.fff files + GPS JSON/CSV
- Required fields: GPS (lat/lon), timestamp, signal file
- Upload via web interface or API
- Support batch submissions (ZIP of .sub files + manifest.json)

#### 2. Data Storage
**Wigle Stats (2017):**
- 349M WiFi networks
- 7.8M cell towers
- Billions of observations
- GPS coordinates for 99%+ of records

**GigLez Architecture:**
- PostgreSQL with PostGIS for geospatial queries
- Deduplicate by file hash + GPS proximity
- Store both raw files and parsed metadata
- Index by frequency, protocol, location, timestamp

#### 3. Search & Discovery
**Wigle Features:**
- Text search (SSID, MAC)
- Geographic search (bounding box, radius)
- Advanced filters (encryption, date range)
- Export to CSV/KML

**GigLez Equivalent:**
- Search by device type, manufacturer, protocol
- Geographic search (radius, bounding box)
- Filter by frequency, modulation, confidence score
- Export to .sub, JSON, CSV, GeoJSON

#### 4. Mapping Interface
**Wigle UI:**
- Zoom-based detail levels
- Color coding by signal type/quality
- Click for network details
- Overlays from entire database

**GigLez UI:**
- Leaflet.js/Mapbox for mapping
- Color by device type or frequency band
- Marker clustering for performance
- Click for device details (.sub file viewer)
- Heatmap overlay for density

#### 5. User Accounts & Gamification
**Wigle System:**
- User registration required
- Upload tracking and statistics
- Leaderboard (global/monthly)
- Contribution milestones

**GigLez System:**
- Optional accounts (allow anonymous uploads)
- Track uploads, identifications, verifications
- Reputation score for accurate IDs
- Badges for contributions (first capture in city, 100 devices, etc.)

#### 6. API Access
**Wigle API:**
- JSON-based RPC (not REST)
- Authentication via API token
- Query endpoints for searching
- Upload endpoints for submissions
- Rate limiting

**GigLez API:**
- RESTful JSON API
- JWT tokens + API keys
- Endpoints:
  - `POST /api/submit` - Upload captures
  - `GET /api/search` - Query database
  - `GET /api/devices/{id}` - Device details
  - `GET /api/heatmap` - Density data
  - `GET /api/stats` - Platform statistics

### Differences from Wigle

| Feature | Wigle.net | GigLez |
|---------|-----------|--------|
| **Data Type** | WiFi, Bluetooth, Cellular | Sub-GHz IoT RF (300-928 MHz) |
| **Submission Format** | CSV | .sub/.fff files + GPS |
| **Identification** | MAC/SSID (exact) | Protocol signature matching (fuzzy) |
| **Focus** | Network mapping | Device type identification |
| **Privacy** | Public by default | Opt-in sharing, anonymization |

## Technical Specifications

### Supported File Formats

#### 1. Flipper Zero .sub Format
```
Filetype: Flipper SubGhz Key File
Version: 1
Frequency: 433920000
Preset: FuriHalSubGhzPresetOok270Async
Protocol: Princeton
Bit: 24
Key: 00 00 00 00 00 95 D5 D4
TE: 400
```

**Key Fields for Matching:**
- `Frequency`: Exact frequency in Hz
- `Preset`: Modulation type (OOK/FSK)
- `Protocol`: Protocol name (if decoded)
- `Bit`: Bit length
- `Key`: Data payload (hex)
- `TE`: Timing element (μs)

#### 2. Flipper RAW Format (.sub)
```
Filetype: Flipper SubGhz RAW File
Version: 1
Frequency: 433920000
Preset: FuriHalSubGhzPresetOok650Async
Protocol: RAW
RAW_Data: 2980 -240 520 -980 520 -980 ...
```

**Parsing Strategy:**
- Extract timing patterns
- Identify repeating sequences
- Match against known protocols by timing signatures
- Calculate pulse width statistics

#### 3. .fff Format (Future Feature)
Support for other firmware formats as needed.

### Submission API Format

#### JSON Manifest
```json
{
  "submission": {
    "timestamp": "2025-01-11T20:30:00Z",
    "latitude": 40.7128,
    "longitude": -74.0060,
    "altitude": 10.5,
    "accuracy": 5.0,
    "session_id": "optional_session_identifier",
    "user_id": "optional_user_identifier",
    "device_name": "Flipper Zero",
    "notes": "Captured near downtown"
  },
  "files": [
    {
      "filename": "capture_001.sub",
      "sha256": "abc123...",
      "size_bytes": 256
    },
    {
      "filename": "capture_002.sub",
      "sha256": "def456...",
      "size_bytes": 312
    }
  ]
}
```

#### CSV Submission Format (Alternative)
```csv
latitude,longitude,timestamp,filename,accuracy,altitude
40.7128,-74.0060,2025-01-11T20:30:00Z,capture_001.sub,5.0,10.5
40.7129,-74.0061,2025-01-11T20:30:15Z,capture_002.sub,5.0,10.5
```

### Device Signature Matching Pipeline

#### Step 1: Parse .sub File
```python
def parse_sub_file(file_path):
    """Extract metadata from .sub file"""
    metadata = {
        'frequency': None,
        'protocol': None,
        'modulation': None,
        'bit_length': None,
        'key_data': None,
        'timing': None,
        'raw_data': None
    }

    with open(file_path) as f:
        for line in f:
            if ':' in line:
                key, value = line.split(':', 1)
                # Map to metadata fields

    return metadata
```

#### Step 2: Match Against Signature Database
```python
def match_signature(metadata):
    """
    Match parsed metadata against known signatures
    Returns: [(device_id, confidence), ...]
    """
    matches = []

    # Exact match: protocol + frequency + bit length
    exact = query_exact_match(
        metadata['protocol'],
        metadata['frequency'],
        metadata['bit_length']
    )
    if exact:
        matches.append((exact.device_id, 1.0))

    # Partial match: protocol + frequency
    partial = query_partial_match(
        metadata['protocol'],
        metadata['frequency']
    )
    for match in partial:
        matches.append((match.device_id, 0.8))

    # Pattern match: bit pattern similarity
    if metadata['key_data']:
        pattern_matches = match_bit_patterns(metadata['key_data'])
        matches.extend(pattern_matches)

    # Timing match: for RAW files
    if metadata['raw_data']:
        timing_matches = match_timing_patterns(metadata['raw_data'])
        matches.extend(timing_matches)

    # Sort by confidence, deduplicate
    return sorted(set(matches), key=lambda x: x[1], reverse=True)
```

#### Step 3: Store Results
```python
def store_capture(file_path, gps_coords, matches):
    """Store capture with matched device(s)"""
    capture = {
        'latitude': gps_coords['lat'],
        'longitude': gps_coords['lon'],
        'timestamp': gps_coords['timestamp'],
        'file_hash': sha256(file_path),
        'file_path': upload_to_storage(file_path),
        **parse_sub_file(file_path)
    }

    # Store capture
    capture_id = db.captures.insert(capture)

    # Store top 3 matches
    for device_id, confidence in matches[:3]:
        db.capture_matches.insert({
            'capture_id': capture_id,
            'device_id': device_id,
            'confidence': confidence,
            'method': 'auto'
        })

    return capture_id
```

## System Architecture

### Platform Components

```
┌─────────────────────────────────────────────────────────┐
│                    Web Interface                         │
│  - Upload Form (drag .sub files + GPS)                  │
│  - Interactive Map (Leaflet.js)                          │
│  - Search & Filter UI                                    │
│  - Device Database Browser                               │
└─────────────────────────────────────────────────────────┘
                           │
                           ↓
┌─────────────────────────────────────────────────────────┐
│                     API Layer (FastAPI)                  │
│  POST /api/submit    - Upload captures                  │
│  GET  /api/search    - Query database                   │
│  GET  /api/devices   - Device catalog                   │
│  GET  /api/heatmap   - Geographic density               │
└─────────────────────────────────────────────────────────┘
                           │
                           ↓
┌─────────────────────────────────────────────────────────┐
│              Processing Pipeline                         │
│  1. File Parser (.sub/.fff → metadata)                  │
│  2. Signature Matcher (metadata → device IDs)           │
│  3. Deduplicator (file hash + GPS proximity)            │
│  4. Storage Manager (DB + file storage)                 │
└─────────────────────────────────────────────────────────┘
                           │
                           ↓
┌─────────────────────────────────────────────────────────┐
│          Database (PostgreSQL + PostGIS)                 │
│  - captures (GPS + metadata + file refs)                │
│  - devices (known device types)                          │
│  - signatures (Flipper/RTL_433 patterns)                │
│  - users (optional accounts)                             │
│  - identifications (community verifications)             │
└─────────────────────────────────────────────────────────┘
                           │
                           ↓
┌─────────────────────────────────────────────────────────┐
│           Signature Databases (Read-Only)                │
│  - Flipper Zero .sub collection (1000+ files)           │
│  - RTL_433 protocol definitions (200+ protocols)        │
│  - Community-contributed signatures                      │
└─────────────────────────────────────────────────────────┘
```

### Key Features

#### Core Platform Features
- **File Upload**: Drag-and-drop .sub files with GPS coordinates
- **Automatic Parsing**: Extract frequency, protocol, modulation, data from files
- **Device Matching**: Identify devices using signature databases
- **Deduplication**: Prevent duplicate submissions via file hashing
- **Geospatial Search**: Find captures near location or within bounding box
- **Heatmap Generation**: Visualize device density by geographic area
- **Export Data**: Download captures as .sub, JSON, CSV, GeoJSON

#### Community Features
- **Manual Identification**: Users can add/correct device IDs
- **Photo Uploads**: Visual evidence of physical devices
- **Voting System**: Upvote/downvote identifications
- **Verification**: High-confidence IDs become verified
- **Contribution Tracking**: Statistics per user
- **Leaderboard**: Top uploaders and verifiers

#### Privacy Features
- **Anonymous Uploads**: No account required
- **GPS Precision Control**: Round coordinates to configurable precision
- **Private Captures**: Opt-out of public database
- **BSSID-style Removal**: Allow device signature removal requests

## Development Phases

### Phase 1: Foundation (Weeks 1-2)
- [x] Database schema design
- [ ] .sub file parser implementation
- [ ] GPS coordinate validation
- [ ] Basic file upload endpoint
- [ ] Storage backend (local/S3)

### Phase 2: Signature Matching (Weeks 3-4)
- [ ] Import Flipper Zero .sub database
- [ ] Import RTL_433 protocol definitions
- [ ] Build matching engine (exact/partial/pattern)
- [ ] Confidence scoring algorithm
- [ ] Match result storage

### Phase 3: Web Interface (Weeks 5-6)
- [ ] Upload form with drag-and-drop
- [ ] Map visualization (Leaflet.js)
- [ ] Search and filter UI
- [ ] Device detail pages
- [ ] Statistics dashboard

### Phase 4: API & Integration (Weeks 7-8)
- [ ] RESTful API endpoints
- [ ] Authentication (JWT/API keys)
- [ ] Rate limiting
- [ ] OpenAPI documentation
- [ ] Client libraries (Python, JS)

### Phase 5: Community Features (Weeks 9-10)
- [ ] User accounts (optional)
- [ ] Manual device identification
- [ ] Photo upload and display
- [ ] Voting system
- [ ] Verification workflow

### Phase 6: Optimization (Weeks 11-12)
- [ ] Database indexing and optimization
- [ ] Caching layer (Redis)
- [ ] CDN for file storage
- [ ] Batch processing queue
- [ ] Materialized view updates

## Success Metrics

### Platform Growth
- Number of unique captures submitted
- Geographic coverage (cities/countries)
- Total .sub files processed
- Database size (captures, devices)

### Community Engagement
- Active users (uploaders + verifiers)
- Manual identifications submitted
- Verification votes cast
- Photo evidence uploads

### Data Quality
- Device identification accuracy (verified/total)
- Average confidence score
- Duplicate detection rate
- Geographic precision distribution

### Technical Performance
- Upload processing time (median)
- Search query latency (p95)
- Map render performance
- API response times

## Technical Constraints

### File Processing
- .sub file size limits (1MB max recommended)
- Batch upload limits (100 files or 50MB per request)
- Supported file formats (.sub initially, .fff future)
- File parsing timeout (5 seconds per file)

### Geographic Data
- GPS coordinate precision (6-8 decimal places)
- Coordinate validation (valid lat/lon ranges)
- Altitude optional (meters above sea level)
- Accuracy metadata (horizontal accuracy in meters)

### Database Scalability
- PostgreSQL with PostGIS for geospatial
- Partitioning by date for large datasets
- Index strategy for common queries
- Materialized views for statistics

### Privacy & Compliance
- GDPR-style data removal
- Optional account system
- GPS anonymization (configurable rounding)
- No PII in .sub file metadata

## Current Implementation Status (Iteration 6/6 Complete)

### Device Identification Architecture

**Final 6-Component Scoring System:**

1. **Timing Analysis** (40% weight) - `src/matcher/timing_analyzer.py`
   - Dual timing validation (SHORT + LONG pulses)
   - Weighted average: SHORT (60%) + LONG (40%)
   - Multi-method extraction (K-means, histogram, percentile)
   - IQR outlier removal (2.5x threshold)
   - Noise tolerance: 15% jitter tested successfully

2. **Preamble Detection** (25% weight) - `src/matcher/preamble_detector.py`
   - 4 detection methods: long_burst, alternating, sync_word, custom
   - Sorted pattern matching (longest first)
   - Preamble boost: +5% for >90% preamble match + >80% overall score
   - Highly discriminative for protocol families

3. **Timing Ratio Matching** (20% weight) - `src/matcher/pattern_decoder.py`
   - Compare LONG/SHORT pulse ratios
   - Highly discriminative (2:1 vs 3:1 separates families)
   - Catches relative timing errors

4. **Frequency Fingerprinting** (10% weight) - `src/matcher/frequency_fingerprint.py`
   - ISM band classification (315/433/868/915 MHz)
   - Tighter tolerance: ±100 kHz (strict), gradual falloff to ±500 kHz
   - Reduces search space from 299 → ~20-30 candidates

5. **Bit Count Matching** (5% weight) - `src/matcher/pattern_decoder.py`
   - Relaxed scoring (unreliable in synthetic signals)
   - Flexible range validation
   - Reduced from 20% weight in iteration 5

6. **Uniqueness Bonus** (0-20% boost) - `src/matcher/pattern_decoder.py`
   - +20% bonus for unique timing (only 1 similar protocol)
   - +15% for 2 similar protocols, +10% for 3, +5% for 4-5
   - Discriminates rare vs. common timing signatures

### Current Accuracy Metrics (Benchmark Results - Iteration 6)

**Test Suite**: 12 synthetic signals across 10 protocols

**Overall Performance:**
- **Top-1 Accuracy**: 33.3% (4/12 correct)
- **Top-3 Accuracy**: 50.0% (6/12 in top 3) ⬆️ **+17% improvement**
- **Top-5 Accuracy**: 50.0%
- **Target**: ≥25% ✅ **PASSED**

**Confidence Distribution:**
- High (>80%): 66.7% (tighter scoring reduces overconfidence)
- Medium (50-80%): 25.0%
- Low (<50%): 8.3%

**Processing Performance:**
- Avg Parse Time: 0.26 ms
- Avg Match Time: 132.81 ms
- **Total**: 133.07 ms per signal (15% faster than iteration 5)

**Key Findings (Iteration 6):**
- ✅ **Noise Tolerance**: Successfully handles 15% jitter
- ✅ **Preamble Detection**: Critical for discrimination (alternating patterns excel)
- ✅ **Timing Ratio**: LONG/SHORT ratio highly discriminative (2:1 vs 3:1)
- ✅ **Family Matching**: Acurite 609TXC → Acurite 896 (correct family, rank 2)
- ✅ **Test Data Fixed**: Corrected Acurite and Oregon timing parameters
- ⚠️ **Princeton**: Still failing (SimpliSafe mismatch) - needs protocol tuning
- ⚠️ **Generic Protocols**: Need more database entries (Nexus, Honeywell, Generic Doorbell not in DB)

### Next Steps for Further Improvement

1. **Add Missing Protocols**: Nexus Temperature-Humidity, Honeywell Security, Generic Doorbell to database
2. **Princeton Tuning**: Investigate SimpliSafe mismatch - may need protocol signature refinement
3. **TPMS Family Logic**: Toyota vs Schrader - add manufacturer-specific heuristics
4. **Real-World Testing**: Benchmark against actual Flipper Zero captures (not synthetic)
5. **Protocol Family Scoring**: Bonus for partial name matches (Acurite 896 vs Acurite 609TXC)


*Last Updated*: 2026-02-15 - Iteration 5/5 complete