Trilltechnician
04bd80b25b
Major milestone: GPS coordinates now auto-extract from filenames and uploads appear on map with full end-to-end workflow functional! ✨ GPS Auto-Extraction Features: - JavaScript GPS extractor class matching Python patterns - Supports 3 filename formats: * N/S/E/W: 34.0478N_118.2349W_filename.sub * lat/lon prefix: lat34.0478lon-118.2348_filename.sub * Signed decimal: -34.0478_118.2348_filename.sub - Auto-populates latitude/longitude form fields on file drop - Green notification toast shows detected coordinates - File list shows GPS badge for files with coordinates 🗺️ Web Interface Improvements: - Upload endpoint now stores captures in-memory - Query endpoint returns uploaded captures for map display - Stats endpoint shows real-time upload counts - Map displays uploaded captures as markers - Color-coded by frequency band 📁 Updated Files: - static/js/upload.js: GPS extraction + auto-population - src/api/main_simple.py: In-memory storage + endpoints - src/parser/gps_extractor.py: Backend GPS extraction (Python) - scripts/test_gps_extraction.py: Python test suite - test_gps_extraction.html: Browser test suite 📊 T-Embed Files Updated: - 34.0478N_118.2348W_1637_raw_8.sub: 315 MHz Princeton - 34.0478N_118.2349W_1351_raw_10.sub: 433.92 MHz Princeton - 34.0478N_118.2349W_1650_test_raw.sub: 433.92 MHz RAW - All now have proper Flipper SubGhz headers ✅ Tested Features: - GPS extraction from filename: 34.0478N_118.2349W → 34.0478, -118.2349 - Auto-population of GPS fields in upload form - File upload with GPS validation - Capture appears on map after upload - Statistics update in real-time - Frequency distribution calculated correctly 🎯 End-to-End Flow Working: 1. User drops .sub file with GPS in filename 2. GPS auto-detected and form fields populate 3. User clicks Upload 4. Server parses RF data + GPS coordinates 5. Capture stored in memory 6. Map refreshes and displays new marker 7. Stats update with new counts 🚀 Demo: http://localhost:8000 Upload 34.0478N_118.2349W_1351_raw_10.sub and watch it appear on map! 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
386 lines
11 KiB
Markdown
386 lines
11 KiB
Markdown
# GigLez Project Status
|
|
|
|
**Date**: 2025-01-11
|
|
**Phase**: Foundation & Planning
|
|
**Status**: Architecture Complete, Ready for Implementation
|
|
|
|
## Completed Tasks
|
|
|
|
### 1. Research & Documentation
|
|
- ✅ Researched Flipper Zero Sub-GHz database structure
|
|
- ✅ Documented RTL_433 protocol database (200+ protocols)
|
|
- ✅ Analyzed Universal Radio Hacker signal formats
|
|
- ✅ Identified key signature sources and file formats
|
|
|
|
### 2. Project Architecture
|
|
- ✅ Created comprehensive CLAUDE.md with primary directives
|
|
- ✅ Designed modular directory structure
|
|
- ✅ Established clear separation of concerns (capture/gps/database/matcher/api/web)
|
|
|
|
### 3. Database Design
|
|
- ✅ Complete PostgreSQL schema with 9+ tables
|
|
- ✅ Geospatial indexing for location queries
|
|
- ✅ Signature matching tables (Flipper, RTL_433, community)
|
|
- ✅ User authentication and community voting system
|
|
- ✅ Materialized views for performance optimization
|
|
- ✅ Trigger functions for automatic statistics updates
|
|
|
|
### 4. Signature Database Integration
|
|
- ✅ Documented Flipper Zero .sub file format parsing
|
|
- ✅ RTL_433 JSON output field mapping
|
|
- ✅ Import pipeline design for all signature sources
|
|
- ✅ Signature matching algorithm specification
|
|
- ✅ Community contribution workflow
|
|
|
|
### 5. Hardware Integration
|
|
- ✅ T-Embed communication protocol design (JSON over serial)
|
|
- ✅ Command reference (SCAN, CAPTURE, STATUS, etc.)
|
|
- ✅ Event notification system
|
|
- ✅ Python controller implementation (sync & async)
|
|
- ✅ GPS coordination strategy
|
|
|
|
### 6. Development Infrastructure
|
|
- ✅ requirements.txt with all dependencies
|
|
- ✅ .gitignore configured for Python/data files
|
|
- ✅ Initial Python module structure
|
|
- ✅ T-Embed controller classes (sync/async)
|
|
|
|
## Current Project Structure
|
|
|
|
```
|
|
giglez/
|
|
├── CLAUDE.md # Primary project directives
|
|
├── README.md # User-facing documentation
|
|
├── PROJECT_STATUS.md # This file
|
|
├── requirements.txt # Python dependencies
|
|
├── .gitignore # Git exclusions
|
|
│
|
|
├── docs/ # Technical documentation
|
|
│ ├── database_schema.md # Complete DB schema
|
|
│ ├── signature_databases.md # Signature import guide
|
|
│ └── tembed_setup.md # Hardware setup & protocol
|
|
│
|
|
├── src/ # Source code
|
|
│ ├── __init__.py
|
|
│ ├── capture/ # T-Embed communication
|
|
│ │ ├── __init__.py
|
|
│ │ ├── tembed.py # Controller implementation
|
|
│ │ └── scanner.py # (TODO) Signal scanner
|
|
│ ├── gps/ # GPS integration
|
|
│ ├── database/ # Database models & ORM
|
|
│ ├── matcher/ # Signature matching engine
|
|
│ ├── api/ # RESTful API
|
|
│ └── web/ # Web interface
|
|
│
|
|
├── signatures/ # Signature databases
|
|
│ ├── flipper/ # Flipper Zero .sub files
|
|
│ ├── rtl433/ # RTL_433 protocols
|
|
│ ├── urh/ # URH signal definitions
|
|
│ └── community/ # User submissions
|
|
│
|
|
├── scripts/ # Utility scripts
|
|
├── config/ # Configuration files
|
|
└── tests/ # Test suite
|
|
```
|
|
|
|
## Key Technical Decisions
|
|
|
|
### 1. Hardware Stack
|
|
- **Primary Device**: LilyGo T-Embed with CC1101 (300-928 MHz)
|
|
- **Control Hub**: Android Termux environment
|
|
- **GPS Source**: Android device native GPS
|
|
- **Communication**: USB Serial (115200 baud, JSON protocol)
|
|
|
|
### 2. Database
|
|
- **Engine**: PostgreSQL with PostGIS for geospatial queries
|
|
- **ORM**: SQLAlchemy for Python integration
|
|
- **Migration**: Alembic for schema versioning
|
|
|
|
### 3. API Architecture
|
|
- **Framework**: FastAPI (async, high performance)
|
|
- **Server**: Uvicorn with uvloop
|
|
- **Authentication**: JWT tokens, API keys
|
|
|
|
### 4. Signature Sources
|
|
1. **Flipper Zero**: 1000+ device signatures (.sub format)
|
|
2. **RTL_433**: 200+ protocol definitions (JSON)
|
|
3. **URH**: Community signal patterns
|
|
4. **User Submissions**: Photos + verified captures
|
|
|
|
### 5. Matching Strategy
|
|
- Exact match: Protocol + Frequency + Timing (100% confidence)
|
|
- Partial match: Protocol + Frequency (80% confidence)
|
|
- Bit pattern matching with masks (90% confidence)
|
|
- Weighted scoring based on source reliability
|
|
|
|
## Next Steps - Implementation Roadmap
|
|
|
|
### Phase 1: Core Infrastructure (Week 1-2)
|
|
```bash
|
|
Priority: HIGH
|
|
```
|
|
|
|
**Database Setup**
|
|
- [ ] Install PostgreSQL in Termux
|
|
- [ ] Create database and user
|
|
- [ ] Run schema creation script (from database_schema.md)
|
|
- [ ] Set up Alembic migrations
|
|
- [ ] Test geospatial queries
|
|
|
|
**T-Embed Integration**
|
|
- [ ] Flash Bruce firmware to T-Embed
|
|
- [ ] Test serial communication from Termux
|
|
- [ ] Verify command/response protocol
|
|
- [ ] Implement error handling and reconnection
|
|
- [ ] Create scanner module (src/capture/scanner.py)
|
|
|
|
**GPS Module**
|
|
- [ ] Create GPS manager (src/gps/manager.py)
|
|
- [ ] Test Android SL4A integration
|
|
- [ ] Implement coordinate streaming
|
|
- [ ] Add accuracy filtering
|
|
- [ ] Create GPS logging
|
|
|
|
### Phase 2: Signature Import (Week 2-3)
|
|
```bash
|
|
Priority: HIGH
|
|
```
|
|
|
|
**Flipper Zero Database**
|
|
- [ ] Clone flipperzero-firmware repository
|
|
- [ ] Extract .sub files from assets
|
|
- [ ] Create parser (src/matcher/flipper_parser.py)
|
|
- [ ] Import signatures to database
|
|
- [ ] Verify protocol coverage
|
|
|
|
**RTL_433 Protocols**
|
|
- [ ] Clone rtl_433 and rtl_433_tests
|
|
- [ ] Extract protocol definitions
|
|
- [ ] Create parser (src/matcher/rtl433_parser.py)
|
|
- [ ] Map JSON fields to database schema
|
|
- [ ] Import test data samples
|
|
|
|
**Import Scripts**
|
|
- [ ] scripts/import_flipper.py
|
|
- [ ] scripts/import_rtl433.py
|
|
- [ ] scripts/update_signatures.sh (cron job)
|
|
|
|
### Phase 3: Matching Engine (Week 3-4)
|
|
```bash
|
|
Priority: HIGH
|
|
```
|
|
|
|
**Signature Matcher**
|
|
- [ ] Create matcher module (src/matcher/engine.py)
|
|
- [ ] Implement exact matching
|
|
- [ ] Implement partial matching
|
|
- [ ] Add bit pattern matching with masks
|
|
- [ ] Create confidence scoring algorithm
|
|
- [ ] Optimize database queries
|
|
|
|
**Testing**
|
|
- [ ] Unit tests for matching logic
|
|
- [ ] Test against known signatures
|
|
- [ ] Benchmark query performance
|
|
- [ ] Validate confidence scores
|
|
|
|
### Phase 4: Capture Workflow (Week 4-5)
|
|
```bash
|
|
Priority: HIGH
|
|
```
|
|
|
|
**Capture Coordination**
|
|
- [ ] Create main capture loop (src/main.py)
|
|
- [ ] Integrate T-Embed + GPS + Database
|
|
- [ ] Implement event handling
|
|
- [ ] Add automatic signature matching
|
|
- [ ] Create session management
|
|
- [ ] Implement capture deduplication (file hashing)
|
|
|
|
**File Management**
|
|
- [ ] .sub file storage on SD card
|
|
- [ ] Automatic file retrieval
|
|
- [ ] Local caching strategy
|
|
- [ ] Export to multiple formats
|
|
|
|
### Phase 5: API Development (Week 5-6)
|
|
```bash
|
|
Priority: MEDIUM
|
|
```
|
|
|
|
**RESTful API** (src/api/)
|
|
- [ ] FastAPI application setup
|
|
- [ ] Authentication (JWT + API keys)
|
|
- [ ] Endpoints:
|
|
- [ ] POST /api/captures (submit capture)
|
|
- [ ] GET /api/captures (query by location/time)
|
|
- [ ] GET /api/devices (device database)
|
|
- [ ] POST /api/identifications (user ID)
|
|
- [ ] GET /api/sessions (capture sessions)
|
|
- [ ] GET /api/heatmap (geographic density)
|
|
- [ ] Rate limiting
|
|
- [ ] CORS configuration
|
|
- [ ] API documentation (OpenAPI/Swagger)
|
|
|
|
### Phase 6: Web Interface (Week 6-8)
|
|
```bash
|
|
Priority: MEDIUM
|
|
```
|
|
|
|
**Mapping UI** (src/web/)
|
|
- [ ] Choose mapping library (Leaflet.js/Mapbox)
|
|
- [ ] Create heatmap visualization
|
|
- [ ] Device marker clustering
|
|
- [ ] Click for device details
|
|
- [ ] Filter by protocol/frequency/device type
|
|
- [ ] Timeline slider for captures
|
|
|
|
**Capture Interface**
|
|
- [ ] Live capture status display
|
|
- [ ] Session controls (start/stop)
|
|
- [ ] Statistics dashboard
|
|
- [ ] Device identification form
|
|
- [ ] Photo upload for evidence
|
|
|
|
**Community Features**
|
|
- [ ] User registration/login
|
|
- [ ] Device identification voting
|
|
- [ ] Reputation system
|
|
- [ ] Leaderboard
|
|
|
|
### Phase 7: Community System (Week 8-10)
|
|
```bash
|
|
Priority: LOW
|
|
```
|
|
|
|
- [ ] User authentication system
|
|
- [ ] Device submission workflow
|
|
- [ ] Photo storage (local/S3)
|
|
- [ ] Voting and verification
|
|
- [ ] Moderation tools
|
|
- [ ] Public API for data access
|
|
|
|
### Phase 8: Optimization & Deployment (Week 10-12)
|
|
```bash
|
|
Priority: LOW
|
|
```
|
|
|
|
- [ ] Database query optimization
|
|
- [ ] Materialized view refresh strategy
|
|
- [ ] Caching layer (Redis)
|
|
- [ ] Background job queue (Celery)
|
|
- [ ] Mobile responsive UI
|
|
- [ ] Docker containerization
|
|
- [ ] CI/CD pipeline
|
|
- [ ] Production deployment guide
|
|
|
|
## Immediate Next Steps (This Week)
|
|
|
|
### 1. Database Setup (Day 1)
|
|
```bash
|
|
# Install PostgreSQL in Termux
|
|
pkg install postgresql
|
|
|
|
# Start PostgreSQL
|
|
initdb ~/postgres
|
|
pg_ctl -D ~/postgres -l logfile start
|
|
|
|
# Create database
|
|
createdb giglez
|
|
|
|
# Run schema
|
|
psql giglez < scripts/schema.sql
|
|
```
|
|
|
|
### 2. Create Schema Script (Day 1)
|
|
Extract SQL from docs/database_schema.md into executable script.
|
|
|
|
### 3. T-Embed Testing (Day 2-3)
|
|
- Flash Bruce firmware
|
|
- Test serial communication
|
|
- Verify JSON protocol
|
|
- Capture test signals
|
|
|
|
### 4. GPS Integration (Day 3-4)
|
|
- Install SL4A in Android
|
|
- Test GPS acquisition
|
|
- Stream coordinates to Termux
|
|
- Log GPS data
|
|
|
|
### 5. First Capture (Day 5)
|
|
- Integrate all components
|
|
- Capture real signal with GPS
|
|
- Store in database
|
|
- Verify data integrity
|
|
|
|
## Known Challenges
|
|
|
|
### Technical
|
|
1. **Battery Life**: Continuous GPS + RF scanning drains battery quickly
|
|
- Solution: Implement duty cycling, power management
|
|
2. **Serial Reliability**: USB serial can disconnect on Android
|
|
- Solution: Auto-reconnect logic, connection monitoring
|
|
3. **Database Size**: Raw captures can grow large quickly
|
|
- Solution: Compression, selective storage, archival strategy
|
|
|
|
### Hardware
|
|
1. **CC1101 Frequency Gaps**: Can't cover entire spectrum
|
|
- 300-348, 387-464, 779-928 MHz only
|
|
2. **Antenna Tuning**: Different frequencies need different antennas
|
|
- Solution: Multi-band antenna or frequency-specific sessions
|
|
|
|
### Community
|
|
1. **Verification Quality**: User-submitted IDs may be incorrect
|
|
- Solution: Multi-voter verification, reputation system
|
|
2. **Privacy**: GPS coordinates could reveal home locations
|
|
- Solution: Anonymization options, GPS precision controls
|
|
|
|
## Resources Needed
|
|
|
|
### Development
|
|
- [ ] Termux on Android with USB OTG support
|
|
- [ ] LilyGo T-Embed CC1101 device
|
|
- [ ] SD card (16GB+) for T-Embed
|
|
- [ ] Multi-band sub-GHz antenna
|
|
|
|
### Testing
|
|
- [ ] Known devices for validation (garage remote, car key fob, etc.)
|
|
- [ ] RTL-SDR for signal verification (optional)
|
|
|
|
## Success Metrics
|
|
|
|
### Phase 1 Complete When:
|
|
- ✅ Database schema created and tested
|
|
- ✅ T-Embed communicates reliably with Termux
|
|
- ✅ GPS coordinates stream to application
|
|
- ✅ First signal captured and stored with GPS
|
|
|
|
### MVP Complete When:
|
|
- ✅ 1000+ signatures imported (Flipper + RTL_433)
|
|
- ✅ Automatic device matching works
|
|
- ✅ Web interface shows captures on map
|
|
- ✅ Can capture, identify, and visualize devices end-to-end
|
|
|
|
### Production Ready When:
|
|
- ✅ Community submission system live
|
|
- ✅ API publicly accessible
|
|
- ✅ 10,000+ captures in database
|
|
- ✅ 50+ verified device types
|
|
- ✅ Multi-user support with authentication
|
|
|
|
## Contributing
|
|
|
|
See [CONTRIBUTING.md](CONTRIBUTING.md) for development workflow, code standards, and pull request process.
|
|
|
|
## Questions & Discussion
|
|
|
|
For questions about architecture decisions or implementation approaches, see:
|
|
- Technical discussions: GitHub Issues
|
|
- Implementation help: GitHub Discussions
|
|
- Real-time chat: [Discord] (coming soon)
|
|
|
|
---
|
|
|
|
**Last Updated**: 2025-01-11
|
|
**Next Review**: After Phase 1 completion
|