giglez/SYSTEM_ANALYSIS.md

GigLez System Analysis - What We've Built

Date: 2026-01-12 Purpose: Complete analysis of implemented features vs. core IoT device identification goal

---

🎯 Core Mission Reminder

Primary Goal: Identify unknown IoT devices from .sub RF captures by matching against signature databases (Flipper Zero, RTL_433) - essentially "Wigle for Sub-GHz IoT devices"

Key Use Case:

User captures unknown signal → Upload .sub file → System identifies:
"This is a Chamberlain garage door opener (433.92 MHz, Rolling Code)"

---

🏗️ What We've Built So Far

Phase 1: Database Infrastructure (100% Complete)

1.1 PostgreSQL Database with PostGIS ✅

Files: scripts/create_schema.sql (800 lines)

What It Does:

• Stores RF captures with GPS coordinates
• Enables geospatial queries (find captures near location)
• Tracks device signatures from multiple sources
• Manages user accounts and sessions

Key Tables:

captures          -- RF signal files with GPS
├── file_hash (PK)        -- SHA256 for deduplication
├── latitude/longitude    -- GPS location
├── frequency            -- RF frequency
├── protocol             -- Decoded protocol (if known)
├── raw_data             -- Timing patterns
└── device_id (FK)       -- Matched device (our goal!)

devices           -- Known IoT device types
├── manufacturer
├── model
├── device_type
├── typical_frequency
└── protocol

signatures        -- Matching patterns
├── device_id (FK)
├── protocol
├── bit_pattern           -- For KEY format
├── bit_mask             -- Which bits to match
└── timing_min/max       -- For RAW format

flipper_signatures -- Flipper Zero database
rtl433_protocols   -- RTL_433 database

Device Identification Tables (Critical for your goal):

• signatures - Protocol patterns to match against
• flipper_signatures - Flipper Zero .sub database
• rtl433_protocols - RTL_433 protocol definitions
• capture_matches - Many-to-many (one capture → multiple possible devices)

Status: Database ready, but signature tables are empty (needs import)

1.2 SQLAlchemy ORM Models ✅

Files: src/database/models.py (600 lines)

What It Does:

• Python classes for database tables
• Automatic GPS geometry population (PostGIS)
• Relationships between tables
• Data validation

Device Identification Models:

Device              # IoT device catalog
Signature           # Matching patterns
CaptureMatch        # Match results (with confidence scores)
FlipperSignature    # Flipper-specific data
RTL433Protocol      # RTL_433-specific data

Status: Models defined, can create/query records

1.3 GPS Validator ✅

Files: src/gps/validator.py (500 lines)

What It Does:

• Validates GPS coordinates (bounds, Null Island, accuracy)
• Anonymization for privacy
• Distance calculations

Not Related to Device ID: This is for data quality, not device matching

---

Phase 2: Upload System (60% Complete)

2.1 FastAPI Application ✅

Files: src/api/main.py (280 lines)

What It Does:

• HTTP API server for uploads and queries
• Environment-aware (dev/production modes)
• CORS, compression, logging middleware
• Health checks

Device ID Relevance: Provides infrastructure for future device matching API

2.2 Storage Abstraction ✅

Files: src/core/storage/*.py (400 lines)

What It Does:

• Saves .sub files (filesystem or S3)
• Content-addressed storage (SHA256 sharding)
• Retrieval and deletion

Not Related to Device ID: Just file storage

2.3 Upload Endpoint ✅

Files: src/api/routes/captures.py (200 lines)

What It Does:

• Accept .sub files with GPS manifest
• Parse .sub file metadata (frequency, protocol, etc.)
• Store in database and filesystem
• Detect duplicates (SHA256)

Critical Gap: Currently does NOT match against device signatures!

Current Flow:

.sub file → Parse metadata → Save to DB → Done
                                          ↑
                                    Missing: Device matching!

Should Be:

.sub file → Parse metadata → Match against signatures →
  → Find best device match → Save with device_id → Done

2.4 .sub File Parser ✅

Files: src/parser/sub_parser.py (300 lines)

What It Does:

• Parses Flipper Zero .sub files (KEY, RAW, BinRAW formats)
• Extracts: frequency, protocol, modulation, bit patterns, timing data

Device ID Relevance: Critical! Extracts features needed for matching

Supported Formats:

1. KEY Format (Decoded):

   Protocol: Princeton
   Frequency: 433920000
   Bit: 24
   Key: 00 00 00 00 00 95 D5 D4  ← Device-specific pattern
   

2. RAW Format (Undecoded):

   Protocol: RAW
   Frequency: 315000000
   RAW_Data: 2980 -240 520 -980...  ← Timing pattern to match
   

3. BinRAW Format (Binary encoded):

   Protocol: BinRAW
   Bit: 48
   Data_RAW: AA AA AA AA AA AA  ← Binary pattern
   

Status: Parser works, extracts features, but not connected to matching engine

---

Phase 2.5: Signature Matching Engine (20% Complete)

2.5.1 Matching Engine Framework ✅

Files: src/matcher/engine.py (121 lines)

What It Does:

• Framework for running multiple matching strategies
• Deduplicates results
• Returns ranked matches with confidence scores

Status: Framework exists but strategies not implemented

2.5.2 Matching Strategies ❌

Files: src/matcher/strategies.py (stub only)

What's Needed (NOT implemented):

class ExactMatchStrategy:
    """Match: protocol + frequency + bit_length"""
    # For KEY format files with decoded protocols

class PartialMatchStrategy:
    """Match: protocol + frequency"""
    # When bit length varies

class BitPatternStrategy:
    """Match bit patterns with masks"""
    # For KEY format: compare key_data against signatures

class TimingPatternStrategy:
    """Match RAW timing patterns"""
    # For RAW format: compare timing sequences

Critical Gap: This is the core functionality that's missing!

---

🧪 What Tests Actually Cover

Implemented Tests ✅

GPS Validator Tests (20+ tests)

File: tests/unit/test_gps_validator.py

What's Tested:

• Valid/invalid coordinates
• Null Island detection
• Accuracy thresholds
• Distance calculations
• Anonymization

Device ID Relevance: None - this is data quality only

Why These Tests Exist: To ensure GPS data is valid before accepting uploads

Not Implemented Tests ❌

Critical Missing Tests:

1. Signature Matching Tests - The core feature!
2. .sub Parser Tests - Verify we extract features correctly
3. Storage Tests - Verify files are saved/retrieved
4. Upload Integration Tests - End-to-end workflow
5. Database Tests - Model operations

---

🔍 Gap Analysis: Device Identification

What's Working ✅

1. Upload .sub files with GPS
2. Parse .sub file metadata
3. Store in database
4. GPS validation
5. Deduplication (SHA256)

What's Missing ❌

1. Signature Database Import (Critical!)

Files Needed:

• scripts/import_flipper.py - Import Flipper Zero signatures
• scripts/import_rtl433.py - Import RTL_433 protocols

What These Do:

# Import Flipper Zero .sub files as signatures
def import_flipper_signatures():
    flipper_repo = clone("flipperzero-firmware")
    for sub_file in flipper_repo.glob("**/*.sub"):
        metadata = parse_sub_file(sub_file)

        # Create device record
        device = Device(
            manufacturer="Unknown",
            model=sub_file.stem,  # Filename
            protocol=metadata.protocol,
            typical_frequency=metadata.frequency
        )

        # Create signature pattern
        signature = Signature(
            device_id=device.id,
            protocol=metadata.protocol,
            frequency=metadata.frequency,
            bit_pattern=metadata.key_data,
            bit_mask=generate_mask(metadata.key_data)
        )

Status: Not implemented - Database has 0 signatures

2. Signature Matching Implementation (Critical!)

Files Needed: src/matcher/strategies.py (implement all strategies)

What These Do:

class ExactMatchStrategy:
    def match(self, metadata, db):
        # For KEY format with decoded protocol
        matches = db.query(Signature).filter(
            Signature.protocol == metadata.protocol,
            Signature.frequency == metadata.frequency,
            Signature.bit_length == metadata.bit_length
        ).all()

        return [MatchResult(
            device_id=sig.device_id,
            confidence=1.0,
            match_method='exact'
        ) for sig in matches]

class TimingPatternStrategy:
    def match(self, metadata, db):
        # For RAW format - compare timing patterns
        raw_timings = metadata.raw_data

        for signature in db.query(Signature).all():
            similarity = compare_timing_patterns(
                raw_timings,
                signature.timing_pattern
            )

            if similarity > 0.8:
                yield MatchResult(
                    device_id=signature.device_id,
                    confidence=similarity,
                    match_method='timing'
                )

Status: Not implemented - No matching happens

3. Background Task Integration ❌

What's Needed: Run matching after upload

@app.post("/api/captures/upload")
async def upload_captures(...):
    # ... existing upload code ...

    # NEW: Match against signatures
    for capture in captures:
        # Run matching in background
        match_task = match_signatures(capture.file_hash)
        background_tasks.add_task(match_task)

Status: Not implemented - Matching not integrated

---

📊 Implementation Status

Infrastructure (Foundation) - 90%

• ✅ Database schema
• ✅ ORM models
• ✅ API server
• ✅ Upload endpoint
• ✅ Storage backend
• ✅ GPS validation
• ✅ .sub parser
• ❌ Configuration (auth, celery)

Core Feature (Device ID) - 5%

• ❌ Signature database import (0%)
• ❌ Matching strategies (0%)
• ✅ Matching framework (100%)
• ❌ Background task integration (0%)
• ❌ Confidence scoring (0%)

Testing - 15%

• ✅ GPS validator tests (100%)
• ❌ Parser tests (0%)
• ❌ Matching tests (0%)
• ❌ Storage tests (0%)
• ❌ Integration tests (0%)

---

🎯 What Needs to Happen for Device ID

Priority 1: Signature Database (Critical)

Without this, matching is impossible

# Step 1: Clone Flipper Zero firmware
git clone https://github.com/flipperdevices/flipperzero-firmware

# Step 2: Import signatures
python scripts/import_flipper.py

# Expected: 1000+ device signatures in database

Priority 2: Implement Matching Strategies (Critical)

This is the core algorithm

For KEY Format (Decoded signals):

def match_key_format(metadata):
    # Exact match: protocol + frequency + bit pattern
    # Use bit masks to ignore variable bits
    # Return confidence score based on match quality

For RAW Format (Unknown signals):

def match_raw_format(metadata):
    # Extract timing pattern from RAW_Data
    # Compare against known timing signatures
    # Use dynamic time warping for similarity
    # Return confidence score

Priority 3: Integration (High)

# After upload, automatically match
capture = save_capture(...)
matches = match_signatures(capture)
if matches:
    capture.device_id = matches[0].device_id
    capture.match_confidence = matches[0].confidence

---

📝 Data Needed for Device Identification

Signature Sources

1. Flipper Zero Database

Source: https://github.com/flipperdevices/flipperzero-firmware Path: applications/main/subghz/assets/*.sub Count: ~200-300 known devices Format: .sub files with known protocols

What We Get:

• Device names (from filename)
• Protocol types (Princeton, KeeLoq, etc.)
• Frequency ranges
• Bit patterns
• Timing characteristics

2. RTL_433 Protocols

Source: https://github.com/merbanan/rtl_433 Path: src/devices/*.c and test files Count: 200+ protocols Format: C code + JSON test data

What We Get:

• Protocol definitions
• Manufacturer names
• Device models
• Modulation types
• Timing specifications

3. Community Contributions

Source: User uploads with manual identification Format: .sub file + photo + verification votes

What We Get:

• Real-world captures
• Visual confirmation
• Geographic distribution

---

🔬 How Matching Would Work

Scenario 1: Decoded Signal (KEY Format)

Input .sub file:
  Protocol: Princeton
  Frequency: 433920000
  Bit: 24
  Key: 00 00 00 00 00 95 D5 D4

Matching Process:
  1. Exact Match: Find signatures with same protocol + frequency
  2. Bit Pattern Match: Compare key_data with bit_mask
  3. Score: 1.0 if exact, 0.8 if partial

Output:
  Device: "Generic 433MHz Remote"
  Manufacturer: "Unknown"
  Confidence: 0.9
  Method: "exact"

Scenario 2: Unknown Signal (RAW Format)

Input .sub file:
  Protocol: RAW
  Frequency: 315000000
  RAW_Data: 2980 -240 520 -980 520 -980...

Matching Process:
  1. Extract Timing Pattern: [2980, 240, 520, 980, ...]
  2. Compare Against Known Patterns (Dynamic Time Warping)
  3. Find Best Match with similarity score

Output:
  Device: "Weather Station (Likely Acurite)"
  Manufacturer: "Acurite"
  Confidence: 0.75
  Method: "timing_pattern"

---

🚦 Current System Flow

What Happens Now (Without Matching)

User → Upload .sub file
     → Parse metadata (protocol, frequency)
     → Save to database (device_id = NULL)
     → Done

What Should Happen (With Matching)

User → Upload .sub file
     → Parse metadata
     → Match against signature database
     → Find best device match (confidence > 0.7)
     → Save to database (device_id = 123, confidence = 0.85)
     → Return: "Chamberlain Garage Door Opener"

---

💡 Recommendations

Immediate Next Steps (Priority Order)

1. Import Flipper Zero Signatures (4-6 hours)

   • Clone repository
   • Write import script
   • Parse .sub files
   • Populate database

2. Implement Exact Matching (2-3 hours)

   • For KEY format files
   • Protocol + frequency + bit pattern
   • Return top 3 matches

3. Test Matching (2 hours)

   • Upload known device
   • Verify correct identification
   • Test confidence scores

4. Implement RAW Matching (6-8 hours)

   • Timing pattern extraction
   • Similarity algorithm
   • Test with unknown signals

5. Background Integration (2 hours)

   • Run matching after upload
   • Update capture with device_id
   • Return results to user

Medium Priority

• Import RTL_433 protocols
• Community identification system
• Voting and verification
• Photo evidence upload

Nice to Have

• Machine learning for unknown signals
• Signal visualization
• Protocol analyzer
• Real-time matching dashboard

---

📈 Progress Toward Core Goal

Infrastructure: 90% ✅

Everything needed to support device identification

Core Feature: 5% ❌

The actual device identification is barely started

Gap: Signature Database + Matching Algorithm

This is what's preventing the system from working

---

🎯 Bottom Line

What We Have:

• Solid infrastructure for uploading, storing, and managing RF captures
• Database designed for device identification
• Parser that extracts features for matching
• Testing framework

What We're Missing:

• Signature database (0 signatures imported)
• Matching algorithms (not implemented)
• Integration (matching not connected to upload)

To Make Device ID Work:

1. Import ~1000 signatures from Flipper Zero
2. Implement matching strategies (exact + timing)
3. Connect matching to upload workflow
4. Test with real .sub files

Time Estimate: 10-15 hours of focused development

---

Current Status: We have a wardriving platform without the device identification engine. Next Focus: Build the matching engine!