Trilltechnician/giglez
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

GPS Auto-Extraction + First Successful Upload Complete

Browse Source 
Major milestone: GPS coordinates now auto-extract from filenames and
uploads appear on map with full end-to-end workflow functional!

✨ GPS Auto-Extraction Features:
- JavaScript GPS extractor class matching Python patterns
- Supports 3 filename formats:
  * N/S/E/W: 34.0478N_118.2349W_filename.sub
  * lat/lon prefix: lat34.0478lon-118.2348_filename.sub
  * Signed decimal: -34.0478_118.2348_filename.sub
- Auto-populates latitude/longitude form fields on file drop
- Green notification toast shows detected coordinates
- File list shows GPS badge for files with coordinates

🗺️ Web Interface Improvements:
- Upload endpoint now stores captures in-memory
- Query endpoint returns uploaded captures for map display
- Stats endpoint shows real-time upload counts
- Map displays uploaded captures as markers
- Color-coded by frequency band

📁 Updated Files:
- static/js/upload.js: GPS extraction + auto-population
- src/api/main_simple.py: In-memory storage + endpoints
- src/parser/gps_extractor.py: Backend GPS extraction (Python)
- scripts/test_gps_extraction.py: Python test suite
- test_gps_extraction.html: Browser test suite

📊 T-Embed Files Updated:
- 34.0478N_118.2348W_1637_raw_8.sub: 315 MHz Princeton
- 34.0478N_118.2349W_1351_raw_10.sub: 433.92 MHz Princeton
- 34.0478N_118.2349W_1650_test_raw.sub: 433.92 MHz RAW
- All now have proper Flipper SubGhz headers

✅ Tested Features:
- GPS extraction from filename: 34.0478N_118.2349W → 34.0478, -118.2349
- Auto-population of GPS fields in upload form
- File upload with GPS validation
- Capture appears on map after upload
- Statistics update in real-time
- Frequency distribution calculated correctly

🎯 End-to-End Flow Working:
1. User drops .sub file with GPS in filename
2. GPS auto-detected and form fields populate
3. User clicks Upload
4. Server parses RF data + GPS coordinates
5. Capture stored in memory
6. Map refreshes and displays new marker
7. Stats update with new counts

🚀 Demo: http://localhost:8000
Upload 34.0478N_118.2349W_1351_raw_10.sub and watch it appear on map!

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
priestlypython
2026-01-13 18:37:13 -08:00
parent 48fcb00241
commit 04bd80b25b
33 changed files with 1903 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/SYSTEM_ANALYSIS.md
+619
View File
@@ -0,0 +1,619 @@
# GigLez System Analysis - What We've Built
**Date**: 2026-01-12
**Purpose**: Complete analysis of implemented features vs. core IoT device identification goal
---
## 🎯 Core Mission Reminder
**Primary Goal**: Identify unknown IoT devices from .sub RF captures by matching against signature databases (Flipper Zero, RTL_433) - essentially "Wigle for Sub-GHz IoT devices"
**Key Use Case**:
```
User captures unknown signal → Upload .sub file → System identifies:
"This is a Chamberlain garage door opener (433.92 MHz, Rolling Code)"
```
---
## 🏗️ What We've Built So Far
### Phase 1: Database Infrastructure (100% Complete)
#### 1.1 PostgreSQL Database with PostGIS ✅
**Files**: `scripts/create_schema.sql` (800 lines)
**What It Does**:
- Stores RF captures with GPS coordinates
- Enables geospatial queries (find captures near location)
- Tracks device signatures from multiple sources
- Manages user accounts and sessions
**Key Tables**:
```sql
captures -- RF signal files with GPS
├── file_hash (PK) -- SHA256 for deduplication
├── latitude/longitude -- GPS location
├── frequency -- RF frequency
├── protocol -- Decoded protocol (if known)
├── raw_data -- Timing patterns
└── device_id (FK) -- Matched device (our goal!)
devices -- Known IoT device types
├── manufacturer
├── model
├── device_type
├── typical_frequency
└── protocol
signatures -- Matching patterns
├── device_id (FK)
├── protocol
├── bit_pattern -- For KEY format
├── bit_mask -- Which bits to match
└── timing_min/max -- For RAW format
flipper_signatures -- Flipper Zero database
rtl433_protocols -- RTL_433 database
```
**Device Identification Tables** (Critical for your goal):
- `signatures` - Protocol patterns to match against
- `flipper_signatures` - Flipper Zero .sub database
- `rtl433_protocols` - RTL_433 protocol definitions
- `capture_matches` - Many-to-many (one capture → multiple possible devices)
**Status**: Database ready, but **signature tables are empty** (needs import)
#### 1.2 SQLAlchemy ORM Models ✅
**Files**: `src/database/models.py` (600 lines)
**What It Does**:
- Python classes for database tables
- Automatic GPS geometry population (PostGIS)
- Relationships between tables
- Data validation
**Device Identification Models**:
```python
Device # IoT device catalog
Signature # Matching patterns
CaptureMatch # Match results (with confidence scores)
FlipperSignature # Flipper-specific data
RTL433Protocol # RTL_433-specific data
```
**Status**: Models defined, can create/query records
#### 1.3 GPS Validator ✅
**Files**: `src/gps/validator.py` (500 lines)
**What It Does**:
- Validates GPS coordinates (bounds, Null Island, accuracy)
- Anonymization for privacy
- Distance calculations
**Not Related to Device ID**: This is for data quality, not device matching
---
### Phase 2: Upload System (60% Complete)
#### 2.1 FastAPI Application ✅
**Files**: `src/api/main.py` (280 lines)
**What It Does**:
- HTTP API server for uploads and queries
- Environment-aware (dev/production modes)
- CORS, compression, logging middleware
- Health checks
**Device ID Relevance**: Provides infrastructure for future device matching API
#### 2.2 Storage Abstraction ✅
**Files**: `src/core/storage/*.py` (400 lines)
**What It Does**:
- Saves .sub files (filesystem or S3)
- Content-addressed storage (SHA256 sharding)
- Retrieval and deletion
**Not Related to Device ID**: Just file storage
#### 2.3 Upload Endpoint ✅
**Files**: `src/api/routes/captures.py` (200 lines)
**What It Does**:
- Accept .sub files with GPS manifest
- Parse .sub file metadata (frequency, protocol, etc.)
- Store in database and filesystem
- Detect duplicates (SHA256)
**Critical Gap**: Currently does **NOT** match against device signatures!
**Current Flow**:
```
.sub file → Parse metadata → Save to DB → Done
↑
Missing: Device matching!
```
**Should Be**:
```
.sub file → Parse metadata → Match against signatures →
→ Find best device match → Save with device_id → Done
```
#### 2.4 .sub File Parser ✅
**Files**: `src/parser/sub_parser.py` (300 lines)
**What It Does**:
- Parses Flipper Zero .sub files (KEY, RAW, BinRAW formats)
- Extracts: frequency, protocol, modulation, bit patterns, timing data
**Device ID Relevance**: **Critical!** Extracts features needed for matching
**Supported Formats**:
1. **KEY Format** (Decoded):
```
Protocol: Princeton
Frequency: 433920000
Bit: 24
Key: 00 00 00 00 00 95 D5 D4 ← Device-specific pattern
```
2. **RAW Format** (Undecoded):
```
Protocol: RAW
Frequency: 315000000
RAW_Data: 2980 -240 520 -980... ← Timing pattern to match
```
3. **BinRAW Format** (Binary encoded):
```
Protocol: BinRAW
Bit: 48
Data_RAW: AA AA AA AA AA AA ← Binary pattern
```
**Status**: Parser works, extracts features, but **not connected to matching engine**
---
### Phase 2.5: Signature Matching Engine (20% Complete)
#### 2.5.1 Matching Engine Framework ✅
**Files**: `src/matcher/engine.py` (121 lines)
**What It Does**:
- Framework for running multiple matching strategies
- Deduplicates results
- Returns ranked matches with confidence scores
**Status**: Framework exists but **strategies not implemented**
#### 2.5.2 Matching Strategies ❌
**Files**: `src/matcher/strategies.py` (stub only)
**What's Needed** (NOT implemented):
```python
class ExactMatchStrategy:
"""Match: protocol + frequency + bit_length"""
# For KEY format files with decoded protocols
class PartialMatchStrategy:
"""Match: protocol + frequency"""
# When bit length varies
class BitPatternStrategy:
"""Match bit patterns with masks"""
# For KEY format: compare key_data against signatures
class TimingPatternStrategy:
"""Match RAW timing patterns"""
# For RAW format: compare timing sequences
```
**Critical Gap**: This is the **core functionality** that's missing!
---
## 🧪 What Tests Actually Cover
### Implemented Tests ✅
#### GPS Validator Tests (20+ tests)
**File**: `tests/unit/test_gps_validator.py`
**What's Tested**:
- Valid/invalid coordinates
- Null Island detection
- Accuracy thresholds
- Distance calculations
- Anonymization
**Device ID Relevance**: None - this is data quality only
**Why These Tests Exist**: To ensure GPS data is valid before accepting uploads
### Not Implemented Tests ❌
**Critical Missing Tests**:
1. **Signature Matching Tests** - The core feature!
2. **.sub Parser Tests** - Verify we extract features correctly
3. **Storage Tests** - Verify files are saved/retrieved
4. **Upload Integration Tests** - End-to-end workflow
5. **Database Tests** - Model operations
---
## 🔍 Gap Analysis: Device Identification
### What's Working ✅
1. Upload .sub files with GPS
2. Parse .sub file metadata
3. Store in database
4. GPS validation
5. Deduplication (SHA256)
### What's Missing ❌
#### 1. Signature Database Import (Critical!)
**Files Needed**:
- `scripts/import_flipper.py` - Import Flipper Zero signatures
- `scripts/import_rtl433.py` - Import RTL_433 protocols
**What These Do**:
```python
# Import Flipper Zero .sub files as signatures
def import_flipper_signatures():
flipper_repo = clone("flipperzero-firmware")
for sub_file in flipper_repo.glob("**/*.sub"):
metadata = parse_sub_file(sub_file)
# Create device record
device = Device(
manufacturer="Unknown",
model=sub_file.stem, # Filename
protocol=metadata.protocol,
typical_frequency=metadata.frequency
)
# Create signature pattern
signature = Signature(
device_id=device.id,
protocol=metadata.protocol,
frequency=metadata.frequency,
bit_pattern=metadata.key_data,
bit_mask=generate_mask(metadata.key_data)
)
```
**Status**: **Not implemented** - Database has 0 signatures
#### 2. Signature Matching Implementation (Critical!)
**Files Needed**: `src/matcher/strategies.py` (implement all strategies)
**What These Do**:
```python
class ExactMatchStrategy:
def match(self, metadata, db):
# For KEY format with decoded protocol
matches = db.query(Signature).filter(
Signature.protocol == metadata.protocol,
Signature.frequency == metadata.frequency,
Signature.bit_length == metadata.bit_length
).all()
return [MatchResult(
device_id=sig.device_id,
confidence=1.0,
match_method='exact'
) for sig in matches]
class TimingPatternStrategy:
def match(self, metadata, db):
# For RAW format - compare timing patterns
raw_timings = metadata.raw_data
for signature in db.query(Signature).all():
similarity = compare_timing_patterns(
raw_timings,
signature.timing_pattern
)
if similarity > 0.8:
yield MatchResult(
device_id=signature.device_id,
confidence=similarity,
match_method='timing'
)
```
**Status**: **Not implemented** - No matching happens
#### 3. Background Task Integration ❌
**What's Needed**: Run matching after upload
```python
@app.post("/api/captures/upload")
async def upload_captures(...):
# ... existing upload code ...
# NEW: Match against signatures
for capture in captures:
# Run matching in background
match_task = match_signatures(capture.file_hash)
background_tasks.add_task(match_task)
```
**Status**: **Not implemented** - Matching not integrated
---
## 📊 Implementation Status
### Infrastructure (Foundation) - 90%
- ✅ Database schema
- ✅ ORM models
- ✅ API server
- ✅ Upload endpoint
- ✅ Storage backend
- ✅ GPS validation
- ✅ .sub parser
- ❌ Configuration (auth, celery)
### Core Feature (Device ID) - 5%
- ❌ Signature database import (0%)
- ❌ Matching strategies (0%)
- ✅ Matching framework (100%)
- ❌ Background task integration (0%)
- ❌ Confidence scoring (0%)
### Testing - 15%
- ✅ GPS validator tests (100%)
- ❌ Parser tests (0%)
- ❌ Matching tests (0%)
- ❌ Storage tests (0%)
- ❌ Integration tests (0%)
---
## 🎯 What Needs to Happen for Device ID
### Priority 1: Signature Database (Critical)
**Without this, matching is impossible**
```bash
# Step 1: Clone Flipper Zero firmware
git clone https://github.com/flipperdevices/flipperzero-firmware
# Step 2: Import signatures
python scripts/import_flipper.py
# Expected: 1000+ device signatures in database
```
### Priority 2: Implement Matching Strategies (Critical)
**This is the core algorithm**
**For KEY Format** (Decoded signals):
```python
def match_key_format(metadata):
# Exact match: protocol + frequency + bit pattern
# Use bit masks to ignore variable bits
# Return confidence score based on match quality
```
**For RAW Format** (Unknown signals):
```python
def match_raw_format(metadata):
# Extract timing pattern from RAW_Data
# Compare against known timing signatures
# Use dynamic time warping for similarity
# Return confidence score
```
### Priority 3: Integration (High)
```python
# After upload, automatically match
capture = save_capture(...)
matches = match_signatures(capture)
if matches:
capture.device_id = matches[0].device_id
capture.match_confidence = matches[0].confidence
```
---
## 📝 Data Needed for Device Identification
### Signature Sources
#### 1. Flipper Zero Database
**Source**: https://github.com/flipperdevices/flipperzero-firmware
**Path**: `applications/main/subghz/assets/*.sub`
**Count**: ~200-300 known devices
**Format**: .sub files with known protocols
**What We Get**:
- Device names (from filename)
- Protocol types (Princeton, KeeLoq, etc.)
- Frequency ranges
- Bit patterns
- Timing characteristics
#### 2. RTL_433 Protocols
**Source**: https://github.com/merbanan/rtl_433
**Path**: `src/devices/*.c` and test files
**Count**: 200+ protocols
**Format**: C code + JSON test data
**What We Get**:
- Protocol definitions
- Manufacturer names
- Device models
- Modulation types
- Timing specifications
#### 3. Community Contributions
**Source**: User uploads with manual identification
**Format**: .sub file + photo + verification votes
**What We Get**:
- Real-world captures
- Visual confirmation
- Geographic distribution
---
## 🔬 How Matching Would Work
### Scenario 1: Decoded Signal (KEY Format)
```
Input .sub file:
Protocol: Princeton
Frequency: 433920000
Bit: 24
Key: 00 00 00 00 00 95 D5 D4
Matching Process:
1. Exact Match: Find signatures with same protocol + frequency
2. Bit Pattern Match: Compare key_data with bit_mask
3. Score: 1.0 if exact, 0.8 if partial
Output:
Device: "Generic 433MHz Remote"
Manufacturer: "Unknown"
Confidence: 0.9
Method: "exact"
```
### Scenario 2: Unknown Signal (RAW Format)
```
Input .sub file:
Protocol: RAW
Frequency: 315000000
RAW_Data: 2980 -240 520 -980 520 -980...
Matching Process:
1. Extract Timing Pattern: [2980, 240, 520, 980, ...]
2. Compare Against Known Patterns (Dynamic Time Warping)
3. Find Best Match with similarity score
Output:
Device: "Weather Station (Likely Acurite)"
Manufacturer: "Acurite"
Confidence: 0.75
Method: "timing_pattern"
```
---
## 🚦 Current System Flow
### What Happens Now (Without Matching)
```
User → Upload .sub file
→ Parse metadata (protocol, frequency)
→ Save to database (device_id = NULL)
→ Done
```
### What Should Happen (With Matching)
```
User → Upload .sub file
→ Parse metadata
→ Match against signature database
→ Find best device match (confidence > 0.7)
→ Save to database (device_id = 123, confidence = 0.85)
→ Return: "Chamberlain Garage Door Opener"
```
---
## 💡 Recommendations
### Immediate Next Steps (Priority Order)
1. **Import Flipper Zero Signatures** (4-6 hours)
- Clone repository
- Write import script
- Parse .sub files
- Populate database
2. **Implement Exact Matching** (2-3 hours)
- For KEY format files
- Protocol + frequency + bit pattern
- Return top 3 matches
3. **Test Matching** (2 hours)
- Upload known device
- Verify correct identification
- Test confidence scores
4. **Implement RAW Matching** (6-8 hours)
- Timing pattern extraction
- Similarity algorithm
- Test with unknown signals
5. **Background Integration** (2 hours)
- Run matching after upload
- Update capture with device_id
- Return results to user
### Medium Priority
- Import RTL_433 protocols
- Community identification system
- Voting and verification
- Photo evidence upload
### Nice to Have
- Machine learning for unknown signals
- Signal visualization
- Protocol analyzer
- Real-time matching dashboard
---
## 📈 Progress Toward Core Goal
### Infrastructure: 90% ✅
Everything needed to support device identification
### Core Feature: 5% ❌
The actual device identification is **barely started**
### Gap: Signature Database + Matching Algorithm
**This is what's preventing the system from working**
---
## 🎯 Bottom Line
**What We Have**:
- Solid infrastructure for uploading, storing, and managing RF captures
- Database designed for device identification
- Parser that extracts features for matching
- Testing framework
**What We're Missing**:
- **Signature database** (0 signatures imported)
- **Matching algorithms** (not implemented)
- **Integration** (matching not connected to upload)
**To Make Device ID Work**:
1. Import ~1000 signatures from Flipper Zero
2. Implement matching strategies (exact + timing)
3. Connect matching to upload workflow
4. Test with real .sub files
**Time Estimate**: 10-15 hours of focused development
---
**Current Status**: We have a wardriving platform without the device identification engine.
**Next Focus**: Build the matching engine!