giglez/docs/SYSTEM_ANALYSIS.md

# GigLez System Analysis - What We've Built

**Date**: 2026-01-12
**Purpose**: Complete analysis of implemented features vs. core IoT device identification goal

---

## 🎯 Core Mission Reminder

**Primary Goal**: Identify unknown IoT devices from .sub RF captures by matching against signature databases (Flipper Zero, RTL_433) - essentially "Wigle for Sub-GHz IoT devices"

**Key Use Case**:
```
User captures unknown signal → Upload .sub file → System identifies:
"This is a Chamberlain garage door opener (433.92 MHz, Rolling Code)"
```

---

## 🏗️ What We've Built So Far

### Phase 1: Database Infrastructure (100% Complete)

#### 1.1 PostgreSQL Database with PostGIS ✅
**Files**: `scripts/create_schema.sql` (800 lines)

**What It Does**:
- Stores RF captures with GPS coordinates
- Enables geospatial queries (find captures near location)
- Tracks device signatures from multiple sources
- Manages user accounts and sessions

**Key Tables**:
```sql
captures          -- RF signal files with GPS
├── file_hash (PK)        -- SHA256 for deduplication
├── latitude/longitude    -- GPS location
├── frequency            -- RF frequency
├── protocol             -- Decoded protocol (if known)
├── raw_data             -- Timing patterns
└── device_id (FK)       -- Matched device (our goal!)

devices           -- Known IoT device types
├── manufacturer
├── model
├── device_type
├── typical_frequency
└── protocol

signatures        -- Matching patterns
├── device_id (FK)
├── protocol
├── bit_pattern           -- For KEY format
├── bit_mask             -- Which bits to match
└── timing_min/max       -- For RAW format

flipper_signatures -- Flipper Zero database
rtl433_protocols   -- RTL_433 database
```

**Device Identification Tables** (Critical for your goal):
- `signatures` - Protocol patterns to match against
- `flipper_signatures` - Flipper Zero .sub database
- `rtl433_protocols` - RTL_433 protocol definitions
- `capture_matches` - Many-to-many (one capture → multiple possible devices)

**Status**: Database ready, but **signature tables are empty** (needs import)

#### 1.2 SQLAlchemy ORM Models ✅
**Files**: `src/database/models.py` (600 lines)

**What It Does**:
- Python classes for database tables
- Automatic GPS geometry population (PostGIS)
- Relationships between tables
- Data validation

**Device Identification Models**:
```python
Device              # IoT device catalog
Signature           # Matching patterns
CaptureMatch        # Match results (with confidence scores)
FlipperSignature    # Flipper-specific data
RTL433Protocol      # RTL_433-specific data
```

**Status**: Models defined, can create/query records

#### 1.3 GPS Validator ✅
**Files**: `src/gps/validator.py` (500 lines)

**What It Does**:
- Validates GPS coordinates (bounds, Null Island, accuracy)
- Anonymization for privacy
- Distance calculations

**Not Related to Device ID**: This is for data quality, not device matching

---

### Phase 2: Upload System (60% Complete)

#### 2.1 FastAPI Application ✅
**Files**: `src/api/main.py` (280 lines)

**What It Does**:
- HTTP API server for uploads and queries
- Environment-aware (dev/production modes)
- CORS, compression, logging middleware
- Health checks

**Device ID Relevance**: Provides infrastructure for future device matching API

#### 2.2 Storage Abstraction ✅
**Files**: `src/core/storage/*.py` (400 lines)

**What It Does**:
- Saves .sub files (filesystem or S3)
- Content-addressed storage (SHA256 sharding)
- Retrieval and deletion

**Not Related to Device ID**: Just file storage

#### 2.3 Upload Endpoint ✅
**Files**: `src/api/routes/captures.py` (200 lines)

**What It Does**:
- Accept .sub files with GPS manifest
- Parse .sub file metadata (frequency, protocol, etc.)
- Store in database and filesystem
- Detect duplicates (SHA256)

**Critical Gap**: Currently does **NOT** match against device signatures!

**Current Flow**:
```
.sub file → Parse metadata → Save to DB → Done
                                          ↑
                                    Missing: Device matching!
```

**Should Be**:
```
.sub file → Parse metadata → Match against signatures →
  → Find best device match → Save with device_id → Done
```

#### 2.4 .sub File Parser ✅
**Files**: `src/parser/sub_parser.py` (300 lines)

**What It Does**:
- Parses Flipper Zero .sub files (KEY, RAW, BinRAW formats)
- Extracts: frequency, protocol, modulation, bit patterns, timing data

**Device ID Relevance**: **Critical!** Extracts features needed for matching

**Supported Formats**:
1. **KEY Format** (Decoded):
   ```
   Protocol: Princeton
   Frequency: 433920000
   Bit: 24
   Key: 00 00 00 00 00 95 D5 D4  ← Device-specific pattern
   ```

2. **RAW Format** (Undecoded):
   ```
   Protocol: RAW
   Frequency: 315000000
   RAW_Data: 2980 -240 520 -980...  ← Timing pattern to match
   ```

3. **BinRAW Format** (Binary encoded):
   ```
   Protocol: BinRAW
   Bit: 48
   Data_RAW: AA AA AA AA AA AA  ← Binary pattern
   ```

**Status**: Parser works, extracts features, but **not connected to matching engine**

---

### Phase 2.5: Signature Matching Engine (20% Complete)

#### 2.5.1 Matching Engine Framework ✅
**Files**: `src/matcher/engine.py` (121 lines)

**What It Does**:
- Framework for running multiple matching strategies
- Deduplicates results
- Returns ranked matches with confidence scores

**Status**: Framework exists but **strategies not implemented**

#### 2.5.2 Matching Strategies ❌
**Files**: `src/matcher/strategies.py` (stub only)

**What's Needed** (NOT implemented):
```python
class ExactMatchStrategy:
    """Match: protocol + frequency + bit_length"""
    # For KEY format files with decoded protocols

class PartialMatchStrategy:
    """Match: protocol + frequency"""
    # When bit length varies

class BitPatternStrategy:
    """Match bit patterns with masks"""
    # For KEY format: compare key_data against signatures

class TimingPatternStrategy:
    """Match RAW timing patterns"""
    # For RAW format: compare timing sequences
```

**Critical Gap**: This is the **core functionality** that's missing!

---

## 🧪 What Tests Actually Cover

### Implemented Tests ✅

#### GPS Validator Tests (20+ tests)
**File**: `tests/unit/test_gps_validator.py`

**What's Tested**:
- Valid/invalid coordinates
- Null Island detection
- Accuracy thresholds
- Distance calculations
- Anonymization

**Device ID Relevance**: None - this is data quality only

**Why These Tests Exist**: To ensure GPS data is valid before accepting uploads

### Not Implemented Tests ❌

**Critical Missing Tests**:
1. **Signature Matching Tests** - The core feature!
2. **.sub Parser Tests** - Verify we extract features correctly
3. **Storage Tests** - Verify files are saved/retrieved
4. **Upload Integration Tests** - End-to-end workflow
5. **Database Tests** - Model operations

---

## 🔍 Gap Analysis: Device Identification

### What's Working ✅
1. Upload .sub files with GPS
2. Parse .sub file metadata
3. Store in database
4. GPS validation
5. Deduplication (SHA256)

### What's Missing ❌

#### 1. Signature Database Import (Critical!)
**Files Needed**:
- `scripts/import_flipper.py` - Import Flipper Zero signatures
- `scripts/import_rtl433.py` - Import RTL_433 protocols

**What These Do**:
```python
# Import Flipper Zero .sub files as signatures
def import_flipper_signatures():
    flipper_repo = clone("flipperzero-firmware")
    for sub_file in flipper_repo.glob("**/*.sub"):
        metadata = parse_sub_file(sub_file)

        # Create device record
        device = Device(
            manufacturer="Unknown",
            model=sub_file.stem,  # Filename
            protocol=metadata.protocol,
            typical_frequency=metadata.frequency
        )

        # Create signature pattern
        signature = Signature(
            device_id=device.id,
            protocol=metadata.protocol,
            frequency=metadata.frequency,
            bit_pattern=metadata.key_data,
            bit_mask=generate_mask(metadata.key_data)
        )
```

**Status**: **Not implemented** - Database has 0 signatures

#### 2. Signature Matching Implementation (Critical!)
**Files Needed**: `src/matcher/strategies.py` (implement all strategies)

**What These Do**:
```python
class ExactMatchStrategy:
    def match(self, metadata, db):
        # For KEY format with decoded protocol
        matches = db.query(Signature).filter(
            Signature.protocol == metadata.protocol,
            Signature.frequency == metadata.frequency,
            Signature.bit_length == metadata.bit_length
        ).all()

        return [MatchResult(
            device_id=sig.device_id,
            confidence=1.0,
            match_method='exact'
        ) for sig in matches]

class TimingPatternStrategy:
    def match(self, metadata, db):
        # For RAW format - compare timing patterns
        raw_timings = metadata.raw_data

        for signature in db.query(Signature).all():
            similarity = compare_timing_patterns(
                raw_timings,
                signature.timing_pattern
            )

            if similarity > 0.8:
                yield MatchResult(
                    device_id=signature.device_id,
                    confidence=similarity,
                    match_method='timing'
                )
```

**Status**: **Not implemented** - No matching happens

#### 3. Background Task Integration ❌
**What's Needed**: Run matching after upload

```python
@app.post("/api/captures/upload")
async def upload_captures(...):
    # ... existing upload code ...

    # NEW: Match against signatures
    for capture in captures:
        # Run matching in background
        match_task = match_signatures(capture.file_hash)
        background_tasks.add_task(match_task)
```

**Status**: **Not implemented** - Matching not integrated

---

## 📊 Implementation Status

### Infrastructure (Foundation) - 90%
- ✅ Database schema
- ✅ ORM models
- ✅ API server
- ✅ Upload endpoint
- ✅ Storage backend
- ✅ GPS validation
- ✅ .sub parser
- ❌ Configuration (auth, celery)

### Core Feature (Device ID) - 5%
- ❌ Signature database import (0%)
- ❌ Matching strategies (0%)
- ✅ Matching framework (100%)
- ❌ Background task integration (0%)
- ❌ Confidence scoring (0%)

### Testing - 15%
- ✅ GPS validator tests (100%)
- ❌ Parser tests (0%)
- ❌ Matching tests (0%)
- ❌ Storage tests (0%)
- ❌ Integration tests (0%)

---

## 🎯 What Needs to Happen for Device ID

### Priority 1: Signature Database (Critical)
**Without this, matching is impossible**

```bash
# Step 1: Clone Flipper Zero firmware
git clone https://github.com/flipperdevices/flipperzero-firmware

# Step 2: Import signatures
python scripts/import_flipper.py

# Expected: 1000+ device signatures in database
```

### Priority 2: Implement Matching Strategies (Critical)
**This is the core algorithm**

**For KEY Format** (Decoded signals):
```python
def match_key_format(metadata):
    # Exact match: protocol + frequency + bit pattern
    # Use bit masks to ignore variable bits
    # Return confidence score based on match quality
```

**For RAW Format** (Unknown signals):
```python
def match_raw_format(metadata):
    # Extract timing pattern from RAW_Data
    # Compare against known timing signatures
    # Use dynamic time warping for similarity
    # Return confidence score
```

### Priority 3: Integration (High)
```python
# After upload, automatically match
capture = save_capture(...)
matches = match_signatures(capture)
if matches:
    capture.device_id = matches[0].device_id
    capture.match_confidence = matches[0].confidence
```

---

## 📝 Data Needed for Device Identification

### Signature Sources

#### 1. Flipper Zero Database
**Source**: https://github.com/flipperdevices/flipperzero-firmware
**Path**: `applications/main/subghz/assets/*.sub`
**Count**: ~200-300 known devices
**Format**: .sub files with known protocols

**What We Get**:
- Device names (from filename)
- Protocol types (Princeton, KeeLoq, etc.)
- Frequency ranges
- Bit patterns
- Timing characteristics

#### 2. RTL_433 Protocols
**Source**: https://github.com/merbanan/rtl_433
**Path**: `src/devices/*.c` and test files
**Count**: 200+ protocols
**Format**: C code + JSON test data

**What We Get**:
- Protocol definitions
- Manufacturer names
- Device models
- Modulation types
- Timing specifications

#### 3. Community Contributions
**Source**: User uploads with manual identification
**Format**: .sub file + photo + verification votes

**What We Get**:
- Real-world captures
- Visual confirmation
- Geographic distribution

---

## 🔬 How Matching Would Work

### Scenario 1: Decoded Signal (KEY Format)
```
Input .sub file:
  Protocol: Princeton
  Frequency: 433920000
  Bit: 24
  Key: 00 00 00 00 00 95 D5 D4

Matching Process:
  1. Exact Match: Find signatures with same protocol + frequency
  2. Bit Pattern Match: Compare key_data with bit_mask
  3. Score: 1.0 if exact, 0.8 if partial

Output:
  Device: "Generic 433MHz Remote"
  Manufacturer: "Unknown"
  Confidence: 0.9
  Method: "exact"
```

### Scenario 2: Unknown Signal (RAW Format)
```
Input .sub file:
  Protocol: RAW
  Frequency: 315000000
  RAW_Data: 2980 -240 520 -980 520 -980...

Matching Process:
  1. Extract Timing Pattern: [2980, 240, 520, 980, ...]
  2. Compare Against Known Patterns (Dynamic Time Warping)
  3. Find Best Match with similarity score

Output:
  Device: "Weather Station (Likely Acurite)"
  Manufacturer: "Acurite"
  Confidence: 0.75
  Method: "timing_pattern"
```

---

## 🚦 Current System Flow

### What Happens Now (Without Matching)
```
User → Upload .sub file
     → Parse metadata (protocol, frequency)
     → Save to database (device_id = NULL)
     → Done
```

### What Should Happen (With Matching)
```
User → Upload .sub file
     → Parse metadata
     → Match against signature database
     → Find best device match (confidence > 0.7)
     → Save to database (device_id = 123, confidence = 0.85)
     → Return: "Chamberlain Garage Door Opener"
```

---

## 💡 Recommendations

### Immediate Next Steps (Priority Order)

1. **Import Flipper Zero Signatures** (4-6 hours)
   - Clone repository
   - Write import script
   - Parse .sub files
   - Populate database

2. **Implement Exact Matching** (2-3 hours)
   - For KEY format files
   - Protocol + frequency + bit pattern
   - Return top 3 matches

3. **Test Matching** (2 hours)
   - Upload known device
   - Verify correct identification
   - Test confidence scores

4. **Implement RAW Matching** (6-8 hours)
   - Timing pattern extraction
   - Similarity algorithm
   - Test with unknown signals

5. **Background Integration** (2 hours)
   - Run matching after upload
   - Update capture with device_id
   - Return results to user

### Medium Priority

- Import RTL_433 protocols
- Community identification system
- Voting and verification
- Photo evidence upload

### Nice to Have

- Machine learning for unknown signals
- Signal visualization
- Protocol analyzer
- Real-time matching dashboard

---

## 📈 Progress Toward Core Goal

### Infrastructure: 90% ✅
Everything needed to support device identification

### Core Feature: 5% ❌
The actual device identification is **barely started**

### Gap: Signature Database + Matching Algorithm
**This is what's preventing the system from working**

---

## 🎯 Bottom Line

**What We Have**:
- Solid infrastructure for uploading, storing, and managing RF captures
- Database designed for device identification
- Parser that extracts features for matching
- Testing framework

**What We're Missing**:
- **Signature database** (0 signatures imported)
- **Matching algorithms** (not implemented)
- **Integration** (matching not connected to upload)

**To Make Device ID Work**:
1. Import ~1000 signatures from Flipper Zero
2. Implement matching strategies (exact + timing)
3. Connect matching to upload workflow
4. Test with real .sub files

**Time Estimate**: 10-15 hours of focused development

---

**Current Status**: We have a wardriving platform without the device identification engine.
**Next Focus**: Build the matching engine!