GPS Auto-Extraction + First Successful Upload Complete
Major milestone: GPS coordinates now auto-extract from filenames and uploads appear on map with full end-to-end workflow functional! ✨ GPS Auto-Extraction Features: - JavaScript GPS extractor class matching Python patterns - Supports 3 filename formats: * N/S/E/W: 34.0478N_118.2349W_filename.sub * lat/lon prefix: lat34.0478lon-118.2348_filename.sub * Signed decimal: -34.0478_118.2348_filename.sub - Auto-populates latitude/longitude form fields on file drop - Green notification toast shows detected coordinates - File list shows GPS badge for files with coordinates 🗺️ Web Interface Improvements: - Upload endpoint now stores captures in-memory - Query endpoint returns uploaded captures for map display - Stats endpoint shows real-time upload counts - Map displays uploaded captures as markers - Color-coded by frequency band 📁 Updated Files: - static/js/upload.js: GPS extraction + auto-population - src/api/main_simple.py: In-memory storage + endpoints - src/parser/gps_extractor.py: Backend GPS extraction (Python) - scripts/test_gps_extraction.py: Python test suite - test_gps_extraction.html: Browser test suite 📊 T-Embed Files Updated: - 34.0478N_118.2348W_1637_raw_8.sub: 315 MHz Princeton - 34.0478N_118.2349W_1351_raw_10.sub: 433.92 MHz Princeton - 34.0478N_118.2349W_1650_test_raw.sub: 433.92 MHz RAW - All now have proper Flipper SubGhz headers ✅ Tested Features: - GPS extraction from filename: 34.0478N_118.2349W → 34.0478, -118.2349 - Auto-population of GPS fields in upload form - File upload with GPS validation - Capture appears on map after upload - Statistics update in real-time - Frequency distribution calculated correctly 🎯 End-to-End Flow Working: 1. User drops .sub file with GPS in filename 2. GPS auto-detected and form fields populate 3. User clicks Upload 4. Server parses RF data + GPS coordinates 5. Capture stored in memory 6. Map refreshes and displays new marker 7. Stats update with new counts 🚀 Demo: http://localhost:8000 Upload 34.0478N_118.2349W_1351_raw_10.sub and watch it appear on map! 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,275 @@
|
||||
# Project Refocus Summary
|
||||
|
||||
**Date**: 2025-01-11
|
||||
**Major Pivot**: Hardware-specific → Platform-agnostic approach
|
||||
|
||||
## What Changed
|
||||
|
||||
### Before: Device-Dependent Architecture
|
||||
- Focused on LilyGo T-Embed integration
|
||||
- Termux + Android GPS as primary setup
|
||||
- Serial communication protocols
|
||||
- Hardware-specific capture workflow
|
||||
|
||||
### After: Platform-Agnostic Web Service
|
||||
- **Accept uploads from ANY capture device**
|
||||
- .sub/.fff file submission with GPS coordinates
|
||||
- Web platform like Wigle.net
|
||||
- Focus on parsing, matching, and visualization
|
||||
|
||||
## Key Principle
|
||||
|
||||
> **We don't care what hardware you use.** If you can generate .sub files with GPS coordinates, you can contribute to GigLez.
|
||||
|
||||
## Core Platform Features
|
||||
|
||||
### 1. File Upload System
|
||||
- **Input**: .sub/.fff files + GPS coordinates
|
||||
- **Submission Methods**:
|
||||
- Web drag-and-drop interface
|
||||
- REST API (`POST /api/submit`)
|
||||
- Batch uploads (ZIP + manifest.json)
|
||||
- **Anonymous or Authenticated**: User's choice
|
||||
|
||||
### 2. Automatic Device Identification
|
||||
|
||||
#### .sub File Parser (`src/parser/`)
|
||||
Extracts metadata from Flipper Zero .sub files:
|
||||
- Frequency, protocol, modulation
|
||||
- Bit length, key data, timing
|
||||
- RAW signal data (timing arrays)
|
||||
- Custom preset configurations
|
||||
|
||||
**Files Created:**
|
||||
- `src/parser/metadata.py` - Data structures for signal metadata
|
||||
- `src/parser/sub_parser.py` - Complete .sub file parser with validation
|
||||
|
||||
#### Signature Matching Engine (`src/matcher/`)
|
||||
Multi-strategy matching against known devices:
|
||||
|
||||
1. **ExactMatcher** (100% confidence)
|
||||
- Protocol + Frequency + Bit Length
|
||||
|
||||
2. **PartialMatcher** (80% confidence)
|
||||
- Protocol + Frequency only
|
||||
|
||||
3. **PatternMatcher** (70-90% confidence)
|
||||
- Bit pattern similarity with masks
|
||||
- Hamming distance calculations
|
||||
|
||||
4. **TimingMatcher** (60-80% confidence)
|
||||
- Pulse timing characteristics from RAW files
|
||||
- Average pulse width matching
|
||||
|
||||
5. **FrequencyMatcher** (50-70% confidence)
|
||||
- Fuzzy frequency matching within tolerance
|
||||
|
||||
**Files Created:**
|
||||
- `src/matcher/engine.py` - Main matching coordinator
|
||||
- `src/matcher/strategies.py` - All matching strategy implementations
|
||||
|
||||
### 3. Wigle.net-Inspired Platform
|
||||
|
||||
#### Analyzed Features
|
||||
|
||||
| Wigle.net Feature | GigLez Implementation |
|
||||
|-------------------|----------------------|
|
||||
| CSV upload format | .sub file upload + GPS JSON/CSV |
|
||||
| 349M+ WiFi networks | Start with IoT RF devices |
|
||||
| Text search (SSID/MAC) | Search by device type/protocol |
|
||||
| Geographic bounding box | PostGIS geospatial queries |
|
||||
| Heatmap visualization | Device density by location |
|
||||
| User leaderboards | Contribution tracking + reputation |
|
||||
| API access | RESTful JSON API |
|
||||
|
||||
#### Key Differences
|
||||
|
||||
- **Wigle**: Exact identification (MAC address = unique)
|
||||
- **GigLez**: Fuzzy matching (signal patterns → likely device)
|
||||
|
||||
- **Wigle**: Public by default
|
||||
- **GigLez**: Opt-in sharing, anonymization options
|
||||
|
||||
## Documentation Updates
|
||||
|
||||
### CLAUDE.md (Primary Directive)
|
||||
Complete rewrite focusing on:
|
||||
- Platform-agnostic submission workflow
|
||||
- Wigle.net analysis and implementation strategy
|
||||
- .sub file parsing and matching pipeline
|
||||
- System architecture diagram
|
||||
- Submission API format specs
|
||||
|
||||
**Removed**: All T-Embed/Termux/hardware-specific references
|
||||
|
||||
### README.md
|
||||
Now emphasizes:
|
||||
- "Platform-agnostic web service"
|
||||
- Supported capture devices (Flipper, HackRF, RTL-SDR, etc.)
|
||||
- Quick start with web upload or API
|
||||
- Submission format examples
|
||||
- Wigle.net comparison table
|
||||
|
||||
**New sections**:
|
||||
- How It Works (diagram)
|
||||
- Supported Capture Devices
|
||||
- API Documentation
|
||||
- Privacy & Security
|
||||
|
||||
## Technical Implementation
|
||||
|
||||
### Completed Components
|
||||
|
||||
#### 1. .sub File Parser
|
||||
```python
|
||||
from src.parser import parse_sub_file
|
||||
|
||||
metadata = parse_sub_file('capture.sub')
|
||||
# Returns: SignalMetadata with frequency, protocol, modulation, etc.
|
||||
```
|
||||
|
||||
**Features**:
|
||||
- Handles KEY, RAW, and BinRAW formats
|
||||
- Extracts timing patterns from RAW files
|
||||
- Validates file structure
|
||||
- Error collection and reporting
|
||||
|
||||
#### 2. Signature Matching Engine
|
||||
```python
|
||||
from src.matcher import SignatureMatcher
|
||||
|
||||
matcher = SignatureMatcher(database)
|
||||
matcher.add_strategy(ExactMatcher())
|
||||
matcher.add_strategy(PartialMatcher())
|
||||
matcher.add_strategy(PatternMatcher())
|
||||
|
||||
matches = matcher.match(metadata, max_results=10)
|
||||
# Returns: List[MatchResult] with confidence scores
|
||||
```
|
||||
|
||||
**Features**:
|
||||
- Pluggable strategy pattern
|
||||
- Automatic deduplication
|
||||
- Confidence-based sorting
|
||||
- Detailed match explanations
|
||||
|
||||
### Database Schema (Already Complete)
|
||||
- `captures` - RF captures with GPS
|
||||
- `devices` - Known device catalog
|
||||
- `signatures` - Matching patterns (Flipper/RTL_433)
|
||||
- `identifications` - Community verifications
|
||||
- PostGIS for geospatial queries
|
||||
|
||||
### Signature Databases (Documented)
|
||||
- **Flipper Zero**: 1000+ .sub files
|
||||
- **RTL_433**: 200+ protocol definitions
|
||||
- **Community**: User-submitted signatures
|
||||
|
||||
Import scripts planned in `scripts/import_signatures.py`
|
||||
|
||||
## Next Steps
|
||||
|
||||
### Phase 1: API Development (Weeks 1-2)
|
||||
```python
|
||||
# FastAPI endpoints needed:
|
||||
POST /api/submit # Upload .sub file + GPS
|
||||
POST /api/submit/batch # ZIP + manifest
|
||||
GET /api/search # Query by location
|
||||
GET /api/devices/{id} # Device details
|
||||
GET /api/heatmap # Density data
|
||||
```
|
||||
|
||||
### Phase 2: Web Interface (Weeks 3-4)
|
||||
- Upload form (drag-and-drop)
|
||||
- Leaflet.js map with markers
|
||||
- Device search and filtering
|
||||
- Statistics dashboard
|
||||
|
||||
### Phase 3: Signature Import (Weeks 5-6)
|
||||
- Clone Flipper Zero firmware repo
|
||||
- Parse and import .sub files
|
||||
- Extract RTL_433 protocol definitions
|
||||
- Build device catalog
|
||||
|
||||
### Phase 4: Community Features (Weeks 7-8)
|
||||
- User accounts (optional)
|
||||
- Manual device identification
|
||||
- Photo uploads
|
||||
- Voting system
|
||||
|
||||
## Benefits of Platform Approach
|
||||
|
||||
### 1. Wider Adoption
|
||||
- ✅ No hardware requirements
|
||||
- ✅ Works with ANY capture device
|
||||
- ✅ Lower barrier to entry
|
||||
- ✅ Focus on data, not devices
|
||||
|
||||
### 2. Community Growth
|
||||
- ✅ Flipper Zero users (largest community)
|
||||
- ✅ RTL-SDR enthusiasts
|
||||
- ✅ Security researchers
|
||||
- ✅ Ham radio operators
|
||||
|
||||
### 3. Data Quality
|
||||
- ✅ Signature matching improves over time
|
||||
- ✅ Community verification
|
||||
- ✅ Multiple sources = better coverage
|
||||
|
||||
### 4. Simplicity
|
||||
- ✅ No device drivers or firmware
|
||||
- ✅ No Android/Termux complexity
|
||||
- ✅ Just upload and go
|
||||
- ✅ Platform handles the rest
|
||||
|
||||
## Migration Notes
|
||||
|
||||
### Files Removed/Obsoleted
|
||||
- `src/capture/tembed.py` - Device-specific controller
|
||||
- `src/gps/` - Android GPS integration (not needed)
|
||||
- `docs/tembed_setup.md` - Hardware setup guide (keep for reference)
|
||||
|
||||
### Files Repurposed
|
||||
- `src/capture/` - Now for file upload handling
|
||||
- `src/gps/` - Now for GPS coordinate validation
|
||||
|
||||
### New Focus Areas
|
||||
1. **Upload Processing**: Handle file uploads efficiently
|
||||
2. **Parsing Pipeline**: Robust .sub file parsing
|
||||
3. **Matching Accuracy**: Improve signature matching
|
||||
4. **Visualization**: Map rendering performance
|
||||
|
||||
## Success Metrics
|
||||
|
||||
### Platform Growth
|
||||
- Unique .sub files submitted
|
||||
- Geographic coverage (cities/countries)
|
||||
- Total devices identified
|
||||
- Community contributors
|
||||
|
||||
### Matching Accuracy
|
||||
- Automatic identification rate
|
||||
- Average confidence score
|
||||
- Manual verification rate
|
||||
- Pattern database growth
|
||||
|
||||
### Community Engagement
|
||||
- User registrations
|
||||
- Manual identifications
|
||||
- Votes cast
|
||||
- Photo submissions
|
||||
|
||||
## Summary
|
||||
|
||||
GigLez is now a **Wigle.net for IoT RF devices** - a crowdsourced platform where anyone can:
|
||||
|
||||
1. Upload .sub files from any capture device
|
||||
2. Get automatic device identification via signature matching
|
||||
3. Visualize IoT device distribution on maps
|
||||
4. Contribute to community knowledge
|
||||
|
||||
**No hardware lock-in. No app required. Just upload and explore.**
|
||||
|
||||
---
|
||||
|
||||
See [CLAUDE.md](CLAUDE.md) for complete technical specifications.
|
||||
Reference in New Issue
Block a user