Trilltechnician/giglez
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

GPS Auto-Extraction + First Successful Upload Complete

Browse Source 
Major milestone: GPS coordinates now auto-extract from filenames and
uploads appear on map with full end-to-end workflow functional!

✨ GPS Auto-Extraction Features:
- JavaScript GPS extractor class matching Python patterns
- Supports 3 filename formats:
  * N/S/E/W: 34.0478N_118.2349W_filename.sub
  * lat/lon prefix: lat34.0478lon-118.2348_filename.sub
  * Signed decimal: -34.0478_118.2348_filename.sub
- Auto-populates latitude/longitude form fields on file drop
- Green notification toast shows detected coordinates
- File list shows GPS badge for files with coordinates

🗺️ Web Interface Improvements:
- Upload endpoint now stores captures in-memory
- Query endpoint returns uploaded captures for map display
- Stats endpoint shows real-time upload counts
- Map displays uploaded captures as markers
- Color-coded by frequency band

📁 Updated Files:
- static/js/upload.js: GPS extraction + auto-population
- src/api/main_simple.py: In-memory storage + endpoints
- src/parser/gps_extractor.py: Backend GPS extraction (Python)
- scripts/test_gps_extraction.py: Python test suite
- test_gps_extraction.html: Browser test suite

📊 T-Embed Files Updated:
- 34.0478N_118.2348W_1637_raw_8.sub: 315 MHz Princeton
- 34.0478N_118.2349W_1351_raw_10.sub: 433.92 MHz Princeton
- 34.0478N_118.2349W_1650_test_raw.sub: 433.92 MHz RAW
- All now have proper Flipper SubGhz headers

✅ Tested Features:
- GPS extraction from filename: 34.0478N_118.2349W → 34.0478, -118.2349
- Auto-population of GPS fields in upload form
- File upload with GPS validation
- Capture appears on map after upload
- Statistics update in real-time
- Frequency distribution calculated correctly

🎯 End-to-End Flow Working:
1. User drops .sub file with GPS in filename
2. GPS auto-detected and form fields populate
3. User clicks Upload
4. Server parses RF data + GPS coordinates
5. Capture stored in memory
6. Map refreshes and displays new marker
7. Stats update with new counts

🚀 Demo: http://localhost:8000
Upload 34.0478N_118.2349W_1351_raw_10.sub and watch it appear on map!

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
priestlypython
2026-01-13 18:37:13 -08:00
parent 48fcb00241
commit 04bd80b25b
33 changed files with 1903 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/README.md
+326
View File
@@ -0,0 +1,326 @@
# GigLez - IoT RF Device Mapping Platform
A Wigle.net-inspired crowdsourced platform for mapping and identifying Sub-GHz IoT RF devices.
## Overview
GigLez is a **platform-agnostic** web service that accepts .sub/.fff file uploads with GPS coordinates, automatically identifies IoT devices using signature databases, and visualizes device distribution on interactive maps. Think of it as Wigle.net for RF IoT devices instead of WiFi networks.
**Key Principle**: We don't care what hardware you use to capture signals. If you can generate .sub files with GPS coordinates, you can contribute to the platform.
## Features
### Core Platform Features
- **File Upload**: Submit .sub/.fff files with GPS coordinates via web or API
- **Automatic Parsing**: Extract frequency, protocol, modulation, and timing from files
- **Device Identification**: Match against 1000+ known signatures (Flipper Zero, RTL_433)
- **Interactive Maps**: Visualize captured devices with heatmaps and clustering
- **Search & Filter**: Query by location, device type, frequency, protocol
- **Anonymous Uploads**: No account required (but optional for tracking contributions)
### Community Features
- **Manual Identification**: Add or correct device IDs with photo evidence
- **Voting System**: Upvote/downvote community identifications
- **Verification**: Earn reputation for accurate identifications
- **Leaderboards**: Track top contributors
- **Data Export**: Download captures as .sub, JSON, CSV, or GeoJSON
## How It Works
```
┌─────────────────┐
│ Capture Device │ (Flipper Zero, LilyGo, RTL-SDR, HackRF, etc.)
│ + GPS Source │
└────────┬────────┘
│
↓ .sub files + GPS coords
┌─────────────────┐
│ GigLez Upload │ (Web form or API)
└────────┬────────┘
│
↓ Parse & Match
┌─────────────────┐
│ Device ID │ (Automatic matching: "Chamberlain Garage Opener")
│ Confidence: 95%│
└────────┬────────┘
│
↓ Store & Display
┌─────────────────┐
│ Interactive │
│ Map View │
└─────────────────┘
```
## Supported Capture Devices
GigLez accepts .sub files from **any** capture device:
- ✅ **Flipper Zero** - Native .sub format
- ✅ **LilyGo T-Embed** - Compatible with Bruce/Marauder firmware
- ✅ **HackRF One** - Convert captures to .sub format
- ✅ **RTL-SDR** - Use rtl_433 + conversion tools
- ✅ **YardStick One** - Convert RfCat captures
- ✅ **Custom Solutions** - Any tool that outputs .sub/.fff format
**Don't have a capture device?** You can still browse and search the community database!
## Quick Start
### Submit Your First Capture
#### Via Web Interface
1. Visit `https://giglez.io/upload`
2. Drag & drop your .sub files
3. Enter GPS coordinates (or upload CSV manifest)
4. (Optional) Create account to track submissions
5. Click Submit - automatic device matching begins!
#### Via API
```bash
curl -X POST https://api.giglez.io/submit \
-H "Content-Type: multipart/form-data" \
-F "file=@capture_001.sub" \
-F "latitude=40.7128" \
-F "longitude=-74.0060" \
-F "timestamp=2025-01-11T20:30:00Z"
```
#### Batch Upload
```bash
# Create manifest.json
{
"captures": [
{
"filename": "capture_001.sub",
"latitude": 40.7128,
"longitude": -74.0060,
"timestamp": "2025-01-11T20:30:00Z"
}
]
}
# Upload ZIP file
curl -X POST https://api.giglez.io/submit/batch \
-F "manifest=@manifest.json" \
-F "archive=@captures.zip"
```
## Submission Format
### Required Fields
- **GPS Coordinates**: Latitude/Longitude (decimal degrees)
- **Timestamp**: ISO 8601 format (e.g., `2025-01-11T20:30:00Z`)
- **.sub/.fff File**: Signal capture file
### Optional Fields
- **Altitude**: Meters above sea level
- **Accuracy**: GPS accuracy in meters
- **Device Name**: What you used to capture (e.g., "Flipper Zero")
- **Notes**: Additional context
- **Photos**: Device identification evidence
### Example .sub File
```
Filetype: Flipper SubGhz Key File
Version: 1
Frequency: 433920000
Preset: FuriHalSubGhzPresetOok270Async
Protocol: Princeton
Bit: 24
Key: 00 00 00 00 00 95 D5 D4
TE: 400
```
## Device Identification
### Automatic Matching
GigLez uses a multi-strategy matching engine:
1. **Exact Match** (100% confidence): Protocol + Frequency + Bit Length
2. **Partial Match** (80% confidence): Protocol + Frequency
3. **Pattern Match** (70-90% confidence): Bit pattern similarity
4. **Timing Match** (60-80% confidence): Pulse timing characteristics
5. **Frequency Match** (50-70% confidence): Frequency proximity
### Signature Databases
- **Flipper Zero Database**: 1000+ device signatures (.sub files)
- **RTL_433 Protocols**: 200+ protocol definitions
- **Community Signatures**: User-submitted verified devices
### Manual Identification
If automatic matching fails or is low confidence, users can:
- Submit device identification with photos
- Vote on other users' identifications
- Earn reputation for verified IDs
## API Documentation
### Authentication
```bash
# Get API token (optional, for tracking submissions)
curl -X POST https://api.giglez.io/auth/register \
-d "email=user@example.com" \
-d "username=wardriver"
# Use token in requests
curl -H "Authorization: Bearer YOUR_TOKEN" \
https://api.giglez.io/api/submit
```
### Endpoints
| Endpoint | Method | Description |
|----------|--------|-------------|
| `/api/submit` | POST | Upload single capture |
| `/api/submit/batch` | POST | Upload multiple captures |
| `/api/search` | GET | Search captures by location/device |
| `/api/devices` | GET | Browse device catalog |
| `/api/devices/{id}` | GET | Device details and captures |
| `/api/heatmap` | GET | Geographic density data |
| `/api/stats` | GET | Platform statistics |
See full documentation at [https://api.giglez.io/docs](https://api.giglez.io/docs)
## Privacy & Security
### GPS Anonymization
- **Precision Control**: Round coordinates to desired precision (default: 10m)
- **Private Mode**: Opt-out of public database
- **Anonymous Uploads**: No account required
### Data Removal
Request removal of your submissions:
```bash
curl -X DELETE https://api.giglez.io/api/captures/{id} \
-H "Authorization: Bearer YOUR_TOKEN"
```
### No PII Collection
- .sub files contain no personally identifiable information
- GPS coordinates are approximate (not exact addresses)
- Optional accounts for contribution tracking only
## Development Setup
### Prerequisites
```bash
# Install PostgreSQL
sudo apt install postgresql postgresql-contrib postgis
# Install Python 3.10+
sudo apt install python3.10 python3-pip
```
### Installation
```bash
# Clone repository
git clone https://github.com/yourusername/giglez.git
cd giglez
# Create virtual environment
python3 -m venv venv
source venv/bin/activate
# Install dependencies
pip install -r requirements.txt
# Initialize database
python scripts/init_db.py
# Import signature databases
python scripts/import_signatures.py
# Run development server
uvicorn src.api.main:app --reload
```
### Project Structure
```
giglez/
├── src/
│ ├── parser/ # .sub file parsing
│ ├── matcher/ # Device signature matching
│ ├── database/ # PostgreSQL models
│ ├── api/ # FastAPI endpoints
│ └── web/ # Web interface
├── signatures/ # Known device signatures
│ ├── flipper/ # Flipper Zero .sub files
│ ├── rtl433/ # RTL_433 protocols
│ └── community/ # User submissions
├── docs/ # Documentation
└── tests/ # Test suite
```
## Wigle.net Comparison
| Feature | Wigle.net | GigLez |
|---------|-----------|--------|
| **Data Type** | WiFi, Bluetooth, Cellular | Sub-GHz IoT RF (300-928 MHz) |
| **Upload Format** | CSV | .sub/.fff files + GPS |
| **Identification** | MAC/SSID (exact) | Signature matching (fuzzy) |
| **Scale** | 349M+ networks | Just getting started! |
| **Focus** | Network mapping | Device type identification |
| **Privacy** | Public by default | Opt-in sharing |
## Contributing
We welcome contributions! See [CONTRIBUTING.md](CONTRIBUTING.md)
### Ways to Contribute
- 📡 **Upload Captures**: Share your .sub files with GPS
- 🔍 **Identify Devices**: Add manual IDs with photos
- 🛠️ **Add Signatures**: Contribute protocol definitions
- 💻 **Code**: Improve matching algorithms, UI, API
- 📖 **Documentation**: Tutorials, guides, translations
## Roadmap
- [x] Platform architecture & database design
- [x] .sub file parser
- [x] Signature matching engine
- [ ] Web upload interface
- [ ] Interactive map visualization
- [ ] API v1 release
- [ ] Mobile app (Android/iOS)
- [ ] Real-time collaboration features
- [ ] Protocol decoder for unknown signals
## License
MIT License - see [LICENSE](LICENSE)
## Acknowledgments
Inspired by:
- [Wigle.net](https://wigle.net) - WiFi/cellular mapping platform
- [Flipper Zero](https://github.com/flipperdevices/flipperzero-firmware) - Sub-GHz signature database
- [RTL_433](https://github.com/merbanan/rtl_433) - Protocol definitions
- The wardriving and RF security community
## Support
- **Documentation**: [docs.giglez.io](https://docs.giglez.io)
- **Issues**: [GitHub Issues](https://github.com/yourusername/giglez/issues)
- **Discord**: [Join our community](https://discord.gg/giglez)
- **Email**: support@giglez.io
---
**⚠️ Legal Notice**: This tool is for research and educational purposes. Always comply with local radio frequency regulations. Capturing RF signals may be regulated in your jurisdiction. GigLez is for passive monitoring only.