276 lines
7.5 KiB
Markdown
276 lines
7.5 KiB
Markdown
# Project Refocus Summary
|
|
|
|
**Date**: 2025-01-11
|
|
**Major Pivot**: Hardware-specific → Platform-agnostic approach
|
|
|
|
## What Changed
|
|
|
|
### Before: Device-Dependent Architecture
|
|
- Focused on LilyGo T-Embed integration
|
|
- Termux + Android GPS as primary setup
|
|
- Serial communication protocols
|
|
- Hardware-specific capture workflow
|
|
|
|
### After: Platform-Agnostic Web Service
|
|
- **Accept uploads from ANY capture device**
|
|
- .sub/.fff file submission with GPS coordinates
|
|
- Web platform like Wigle.net
|
|
- Focus on parsing, matching, and visualization
|
|
|
|
## Key Principle
|
|
|
|
> **We don't care what hardware you use.** If you can generate .sub files with GPS coordinates, you can contribute to GigLez.
|
|
|
|
## Core Platform Features
|
|
|
|
### 1. File Upload System
|
|
- **Input**: .sub/.fff files + GPS coordinates
|
|
- **Submission Methods**:
|
|
- Web drag-and-drop interface
|
|
- REST API (`POST /api/submit`)
|
|
- Batch uploads (ZIP + manifest.json)
|
|
- **Anonymous or Authenticated**: User's choice
|
|
|
|
### 2. Automatic Device Identification
|
|
|
|
#### .sub File Parser (`src/parser/`)
|
|
Extracts metadata from Flipper Zero .sub files:
|
|
- Frequency, protocol, modulation
|
|
- Bit length, key data, timing
|
|
- RAW signal data (timing arrays)
|
|
- Custom preset configurations
|
|
|
|
**Files Created:**
|
|
- `src/parser/metadata.py` - Data structures for signal metadata
|
|
- `src/parser/sub_parser.py` - Complete .sub file parser with validation
|
|
|
|
#### Signature Matching Engine (`src/matcher/`)
|
|
Multi-strategy matching against known devices:
|
|
|
|
1. **ExactMatcher** (100% confidence)
|
|
- Protocol + Frequency + Bit Length
|
|
|
|
2. **PartialMatcher** (80% confidence)
|
|
- Protocol + Frequency only
|
|
|
|
3. **PatternMatcher** (70-90% confidence)
|
|
- Bit pattern similarity with masks
|
|
- Hamming distance calculations
|
|
|
|
4. **TimingMatcher** (60-80% confidence)
|
|
- Pulse timing characteristics from RAW files
|
|
- Average pulse width matching
|
|
|
|
5. **FrequencyMatcher** (50-70% confidence)
|
|
- Fuzzy frequency matching within tolerance
|
|
|
|
**Files Created:**
|
|
- `src/matcher/engine.py` - Main matching coordinator
|
|
- `src/matcher/strategies.py` - All matching strategy implementations
|
|
|
|
### 3. Wigle.net-Inspired Platform
|
|
|
|
#### Analyzed Features
|
|
|
|
| Wigle.net Feature | GigLez Implementation |
|
|
|-------------------|----------------------|
|
|
| CSV upload format | .sub file upload + GPS JSON/CSV |
|
|
| 349M+ WiFi networks | Start with IoT RF devices |
|
|
| Text search (SSID/MAC) | Search by device type/protocol |
|
|
| Geographic bounding box | PostGIS geospatial queries |
|
|
| Heatmap visualization | Device density by location |
|
|
| User leaderboards | Contribution tracking + reputation |
|
|
| API access | RESTful JSON API |
|
|
|
|
#### Key Differences
|
|
|
|
- **Wigle**: Exact identification (MAC address = unique)
|
|
- **GigLez**: Fuzzy matching (signal patterns → likely device)
|
|
|
|
- **Wigle**: Public by default
|
|
- **GigLez**: Opt-in sharing, anonymization options
|
|
|
|
## Documentation Updates
|
|
|
|
### CLAUDE.md (Primary Directive)
|
|
Complete rewrite focusing on:
|
|
- Platform-agnostic submission workflow
|
|
- Wigle.net analysis and implementation strategy
|
|
- .sub file parsing and matching pipeline
|
|
- System architecture diagram
|
|
- Submission API format specs
|
|
|
|
**Removed**: All T-Embed/Termux/hardware-specific references
|
|
|
|
### README.md
|
|
Now emphasizes:
|
|
- "Platform-agnostic web service"
|
|
- Supported capture devices (Flipper, HackRF, RTL-SDR, etc.)
|
|
- Quick start with web upload or API
|
|
- Submission format examples
|
|
- Wigle.net comparison table
|
|
|
|
**New sections**:
|
|
- How It Works (diagram)
|
|
- Supported Capture Devices
|
|
- API Documentation
|
|
- Privacy & Security
|
|
|
|
## Technical Implementation
|
|
|
|
### Completed Components
|
|
|
|
#### 1. .sub File Parser
|
|
```python
|
|
from src.parser import parse_sub_file
|
|
|
|
metadata = parse_sub_file('capture.sub')
|
|
# Returns: SignalMetadata with frequency, protocol, modulation, etc.
|
|
```
|
|
|
|
**Features**:
|
|
- Handles KEY, RAW, and BinRAW formats
|
|
- Extracts timing patterns from RAW files
|
|
- Validates file structure
|
|
- Error collection and reporting
|
|
|
|
#### 2. Signature Matching Engine
|
|
```python
|
|
from src.matcher import SignatureMatcher
|
|
|
|
matcher = SignatureMatcher(database)
|
|
matcher.add_strategy(ExactMatcher())
|
|
matcher.add_strategy(PartialMatcher())
|
|
matcher.add_strategy(PatternMatcher())
|
|
|
|
matches = matcher.match(metadata, max_results=10)
|
|
# Returns: List[MatchResult] with confidence scores
|
|
```
|
|
|
|
**Features**:
|
|
- Pluggable strategy pattern
|
|
- Automatic deduplication
|
|
- Confidence-based sorting
|
|
- Detailed match explanations
|
|
|
|
### Database Schema (Already Complete)
|
|
- `captures` - RF captures with GPS
|
|
- `devices` - Known device catalog
|
|
- `signatures` - Matching patterns (Flipper/RTL_433)
|
|
- `identifications` - Community verifications
|
|
- PostGIS for geospatial queries
|
|
|
|
### Signature Databases (Documented)
|
|
- **Flipper Zero**: 1000+ .sub files
|
|
- **RTL_433**: 200+ protocol definitions
|
|
- **Community**: User-submitted signatures
|
|
|
|
Import scripts planned in `scripts/import_signatures.py`
|
|
|
|
## Next Steps
|
|
|
|
### Phase 1: API Development (Weeks 1-2)
|
|
```python
|
|
# FastAPI endpoints needed:
|
|
POST /api/submit # Upload .sub file + GPS
|
|
POST /api/submit/batch # ZIP + manifest
|
|
GET /api/search # Query by location
|
|
GET /api/devices/{id} # Device details
|
|
GET /api/heatmap # Density data
|
|
```
|
|
|
|
### Phase 2: Web Interface (Weeks 3-4)
|
|
- Upload form (drag-and-drop)
|
|
- Leaflet.js map with markers
|
|
- Device search and filtering
|
|
- Statistics dashboard
|
|
|
|
### Phase 3: Signature Import (Weeks 5-6)
|
|
- Clone Flipper Zero firmware repo
|
|
- Parse and import .sub files
|
|
- Extract RTL_433 protocol definitions
|
|
- Build device catalog
|
|
|
|
### Phase 4: Community Features (Weeks 7-8)
|
|
- User accounts (optional)
|
|
- Manual device identification
|
|
- Photo uploads
|
|
- Voting system
|
|
|
|
## Benefits of Platform Approach
|
|
|
|
### 1. Wider Adoption
|
|
- ✅ No hardware requirements
|
|
- ✅ Works with ANY capture device
|
|
- ✅ Lower barrier to entry
|
|
- ✅ Focus on data, not devices
|
|
|
|
### 2. Community Growth
|
|
- ✅ Flipper Zero users (largest community)
|
|
- ✅ RTL-SDR enthusiasts
|
|
- ✅ Security researchers
|
|
- ✅ Ham radio operators
|
|
|
|
### 3. Data Quality
|
|
- ✅ Signature matching improves over time
|
|
- ✅ Community verification
|
|
- ✅ Multiple sources = better coverage
|
|
|
|
### 4. Simplicity
|
|
- ✅ No device drivers or firmware
|
|
- ✅ No Android/Termux complexity
|
|
- ✅ Just upload and go
|
|
- ✅ Platform handles the rest
|
|
|
|
## Migration Notes
|
|
|
|
### Files Removed/Obsoleted
|
|
- `src/capture/tembed.py` - Device-specific controller
|
|
- `src/gps/` - Android GPS integration (not needed)
|
|
- `docs/tembed_setup.md` - Hardware setup guide (keep for reference)
|
|
|
|
### Files Repurposed
|
|
- `src/capture/` - Now for file upload handling
|
|
- `src/gps/` - Now for GPS coordinate validation
|
|
|
|
### New Focus Areas
|
|
1. **Upload Processing**: Handle file uploads efficiently
|
|
2. **Parsing Pipeline**: Robust .sub file parsing
|
|
3. **Matching Accuracy**: Improve signature matching
|
|
4. **Visualization**: Map rendering performance
|
|
|
|
## Success Metrics
|
|
|
|
### Platform Growth
|
|
- Unique .sub files submitted
|
|
- Geographic coverage (cities/countries)
|
|
- Total devices identified
|
|
- Community contributors
|
|
|
|
### Matching Accuracy
|
|
- Automatic identification rate
|
|
- Average confidence score
|
|
- Manual verification rate
|
|
- Pattern database growth
|
|
|
|
### Community Engagement
|
|
- User registrations
|
|
- Manual identifications
|
|
- Votes cast
|
|
- Photo submissions
|
|
|
|
## Summary
|
|
|
|
GigLez is now a **Wigle.net for IoT RF devices** - a crowdsourced platform where anyone can:
|
|
|
|
1. Upload .sub files from any capture device
|
|
2. Get automatic device identification via signature matching
|
|
3. Visualize IoT device distribution on maps
|
|
4. Contribute to community knowledge
|
|
|
|
**No hardware lock-in. No app required. Just upload and explore.**
|
|
|
|
---
|
|
|
|
See [CLAUDE.md](CLAUDE.md) for complete technical specifications.
|