GPS Auto-Extraction + First Successful Upload Complete
Major milestone: GPS coordinates now auto-extract from filenames and uploads appear on map with full end-to-end workflow functional! ✨ GPS Auto-Extraction Features: - JavaScript GPS extractor class matching Python patterns - Supports 3 filename formats: * N/S/E/W: 34.0478N_118.2349W_filename.sub * lat/lon prefix: lat34.0478lon-118.2348_filename.sub * Signed decimal: -34.0478_118.2348_filename.sub - Auto-populates latitude/longitude form fields on file drop - Green notification toast shows detected coordinates - File list shows GPS badge for files with coordinates 🗺️ Web Interface Improvements: - Upload endpoint now stores captures in-memory - Query endpoint returns uploaded captures for map display - Stats endpoint shows real-time upload counts - Map displays uploaded captures as markers - Color-coded by frequency band 📁 Updated Files: - static/js/upload.js: GPS extraction + auto-population - src/api/main_simple.py: In-memory storage + endpoints - src/parser/gps_extractor.py: Backend GPS extraction (Python) - scripts/test_gps_extraction.py: Python test suite - test_gps_extraction.html: Browser test suite 📊 T-Embed Files Updated: - 34.0478N_118.2348W_1637_raw_8.sub: 315 MHz Princeton - 34.0478N_118.2349W_1351_raw_10.sub: 433.92 MHz Princeton - 34.0478N_118.2349W_1650_test_raw.sub: 433.92 MHz RAW - All now have proper Flipper SubGhz headers ✅ Tested Features: - GPS extraction from filename: 34.0478N_118.2349W → 34.0478, -118.2349 - Auto-population of GPS fields in upload form - File upload with GPS validation - Capture appears on map after upload - Statistics update in real-time - Frequency distribution calculated correctly 🎯 End-to-End Flow Working: 1. User drops .sub file with GPS in filename 2. GPS auto-detected and form fields populate 3. User clicks Upload 4. Server parses RF data + GPS coordinates 5. Capture stored in memory 6. Map refreshes and displays new marker 7. Stats update with new counts 🚀 Demo: http://localhost:8000 Upload 34.0478N_118.2349W_1351_raw_10.sub and watch it appear on map! 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,360 @@
|
||||
# GigLez Database Setup Guide
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- ✅ PostgreSQL 16+ installed
|
||||
- ✅ PostGIS 3.4+ extension available
|
||||
- ✅ Python 3.8+ with dependencies from requirements.txt
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Step 1: Run Database Setup Script
|
||||
|
||||
```bash
|
||||
cd /home/dell/coding/giglez
|
||||
./scripts/setup_database.sh
|
||||
```
|
||||
|
||||
This script will:
|
||||
1. Create PostgreSQL user `giglez_user`
|
||||
2. Create database `giglez`
|
||||
3. Enable PostGIS extension
|
||||
4. Grant necessary permissions
|
||||
|
||||
**Default Credentials:**
|
||||
- **User**: giglez_user
|
||||
- **Password**: giglez_secure_password_2026
|
||||
- **Database**: giglez
|
||||
- **Host**: localhost
|
||||
- **Port**: 5432
|
||||
|
||||
⚠️ **Security Note**: Change the password in production!
|
||||
|
||||
### Step 2: Create Database Schema
|
||||
|
||||
```bash
|
||||
psql -U giglez_user -d giglez -h localhost -f scripts/create_schema.sql
|
||||
```
|
||||
|
||||
Or if you have sudo access:
|
||||
|
||||
```bash
|
||||
sudo -u postgres psql -d giglez -f scripts/create_schema.sql
|
||||
```
|
||||
|
||||
This will create:
|
||||
- ✅ 15 tables (captures, devices, sessions, signatures, etc.)
|
||||
- ✅ PostGIS geometry columns and spatial indexes
|
||||
- ✅ Materialized views for performance
|
||||
- ✅ Triggers for automatic statistics updates
|
||||
- ✅ Functions for geospatial calculations
|
||||
|
||||
### Step 3: Configure Environment
|
||||
|
||||
```bash
|
||||
# Copy environment template
|
||||
cp .env.example .env
|
||||
|
||||
# Edit with your configuration
|
||||
nano .env
|
||||
```
|
||||
|
||||
Update these values:
|
||||
```env
|
||||
GIGLEZ_DB_PASSWORD=your_secure_password_here
|
||||
GIGLEZ_STORAGE_PATH=/your/storage/path
|
||||
```
|
||||
|
||||
### Step 4: Test Connection
|
||||
|
||||
```bash
|
||||
python3 config/database.py
|
||||
```
|
||||
|
||||
Expected output:
|
||||
```
|
||||
Testing database connection...
|
||||
✅ Database connection test successful
|
||||
Testing PostGIS extension...
|
||||
✅ PostGIS available: 3.4 USE_GEOS=1 USE_PROJ=1 USE_STATS=1
|
||||
✅ Database fully operational
|
||||
```
|
||||
|
||||
## Database Schema Overview
|
||||
|
||||
### Core Tables
|
||||
|
||||
#### captures
|
||||
Primary storage for RF signal captures with GPS coordinates.
|
||||
- **Primary Key**: `file_hash` (SHA256 of .sub file)
|
||||
- **Geospatial**: `geom` column with GIST index
|
||||
- **Deduplication**: Automatic via primary key constraint
|
||||
|
||||
#### devices
|
||||
Known IoT device types from signature databases.
|
||||
- Manufacturer, model, device type
|
||||
- RF characteristics (frequency, modulation, protocol)
|
||||
- Full-text search enabled
|
||||
|
||||
#### sessions
|
||||
Wardriving session grouping (Wigle pattern).
|
||||
- Groups related captures
|
||||
- Tracks statistics (total captures, unique devices)
|
||||
- Bounding box for geographic extent
|
||||
|
||||
#### signatures
|
||||
Protocol signatures for device matching.
|
||||
- Bit patterns with masks
|
||||
- Timing patterns for RAW signals
|
||||
- Confidence weighting
|
||||
|
||||
#### capture_matches
|
||||
Many-to-many mapping of captures to devices.
|
||||
- Multiple devices can match one capture
|
||||
- Confidence scores (0.0 - 1.0)
|
||||
- Match method tracking
|
||||
|
||||
### Supporting Tables
|
||||
|
||||
- **users**: Optional user accounts
|
||||
- **identifications**: Community device submissions
|
||||
- **votes**: Voting on identifications
|
||||
- **upload_markers**: Incremental sync tracking (Wigle pattern)
|
||||
- **flipper_signatures**: Flipper Zero specific data
|
||||
- **rtl433_protocols**: RTL_433 specific data
|
||||
|
||||
### Materialized Views
|
||||
|
||||
#### device_statistics
|
||||
Pre-computed device statistics for performance.
|
||||
```sql
|
||||
REFRESH MATERIALIZED VIEW CONCURRENTLY device_statistics;
|
||||
```
|
||||
|
||||
#### geographic_heatmap
|
||||
Aggregated capture density by location.
|
||||
```sql
|
||||
REFRESH MATERIALIZED VIEW CONCURRENTLY geographic_heatmap;
|
||||
```
|
||||
|
||||
## Common Operations
|
||||
|
||||
### Query Captures Near Location
|
||||
|
||||
```sql
|
||||
-- Within 1km radius
|
||||
SELECT c.*, d.manufacturer, d.model
|
||||
FROM captures c
|
||||
LEFT JOIN devices d ON c.device_id = d.id
|
||||
WHERE calculate_distance(c.latitude, c.longitude, 40.7128, -74.0060) <= 1.0
|
||||
ORDER BY c.captured_at DESC;
|
||||
|
||||
-- Or using PostGIS (faster)
|
||||
SELECT c.*, d.manufacturer, d.model
|
||||
FROM captures c
|
||||
LEFT JOIN devices d ON c.device_id = d.id
|
||||
WHERE ST_DWithin(
|
||||
c.geom,
|
||||
ST_SetSRID(ST_MakePoint(-74.0060, 40.7128), 4326)::geography,
|
||||
1000 -- meters
|
||||
)
|
||||
ORDER BY c.captured_at DESC;
|
||||
```
|
||||
|
||||
### Query by Bounding Box
|
||||
|
||||
```sql
|
||||
SELECT c.*
|
||||
FROM captures c
|
||||
WHERE c.geom && ST_MakeEnvelope(-74.1, 40.6, -73.9, 40.8, 4326)
|
||||
LIMIT 100;
|
||||
```
|
||||
|
||||
### Get Unidentified Captures
|
||||
|
||||
```sql
|
||||
SELECT c.file_hash, c.frequency, c.protocol, c.latitude, c.longitude
|
||||
FROM captures c
|
||||
WHERE c.device_id IS NULL
|
||||
AND c.protocol IS NOT NULL
|
||||
ORDER BY c.captured_at DESC
|
||||
LIMIT 100;
|
||||
```
|
||||
|
||||
### Top Devices by Capture Count
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
d.manufacturer,
|
||||
d.model,
|
||||
d.device_type,
|
||||
COUNT(c.file_hash) as captures
|
||||
FROM devices d
|
||||
JOIN captures c ON c.device_id = d.id
|
||||
GROUP BY d.id, d.manufacturer, d.model, d.device_type
|
||||
ORDER BY captures DESC
|
||||
LIMIT 20;
|
||||
```
|
||||
|
||||
### Heatmap Data
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
lat_bucket,
|
||||
lon_bucket,
|
||||
capture_count,
|
||||
unique_devices
|
||||
FROM geographic_heatmap
|
||||
WHERE capture_count > 5
|
||||
ORDER BY capture_count DESC
|
||||
LIMIT 1000;
|
||||
```
|
||||
|
||||
## Performance Tuning
|
||||
|
||||
### Indexes
|
||||
|
||||
All critical indexes are created automatically:
|
||||
- Geospatial: GIST indexes on `geom` column
|
||||
- Foreign keys: B-tree indexes
|
||||
- Query fields: Indexes on frequency, protocol, timestamp
|
||||
|
||||
### Batch Inserts
|
||||
|
||||
Use transaction batching for bulk inserts (Wigle pattern):
|
||||
|
||||
```python
|
||||
from config.database import DatabaseSession
|
||||
|
||||
BATCH_SIZE = 512 # Wigle optimal batch size
|
||||
|
||||
with DatabaseSession() as session:
|
||||
for i in range(0, len(captures), BATCH_SIZE):
|
||||
batch = captures[i:i+BATCH_SIZE]
|
||||
session.bulk_insert_mappings(Capture, batch)
|
||||
```
|
||||
|
||||
### Connection Pooling
|
||||
|
||||
Configured in `config/database.py`:
|
||||
- Pool size: 10 connections
|
||||
- Max overflow: 20 connections
|
||||
- Pre-ping: True (verify before use)
|
||||
|
||||
### Materialized View Refresh
|
||||
|
||||
Set up cron job for periodic refresh:
|
||||
|
||||
```bash
|
||||
# Add to crontab
|
||||
0 */6 * * * psql -U giglez_user -d giglez -c "REFRESH MATERIALIZED VIEW CONCURRENTLY device_statistics;"
|
||||
0 */6 * * * psql -U giglez_user -d giglez -c "REFRESH MATERIALIZED VIEW CONCURRENTLY geographic_heatmap;"
|
||||
```
|
||||
|
||||
## Backup & Maintenance
|
||||
|
||||
### Backup Database
|
||||
|
||||
```bash
|
||||
pg_dump -U giglez_user -d giglez -h localhost -F c -f giglez_backup_$(date +%Y%m%d).dump
|
||||
```
|
||||
|
||||
### Restore Database
|
||||
|
||||
```bash
|
||||
pg_restore -U giglez_user -d giglez -h localhost giglez_backup_20260112.dump
|
||||
```
|
||||
|
||||
### Vacuum and Analyze
|
||||
|
||||
```bash
|
||||
psql -U giglez_user -d giglez -h localhost -c "VACUUM ANALYZE;"
|
||||
```
|
||||
|
||||
### Check Database Size
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
pg_size_pretty(pg_database_size('giglez')) as db_size,
|
||||
pg_size_pretty(pg_total_relation_size('captures')) as captures_size,
|
||||
pg_size_pretty(pg_total_relation_size('devices')) as devices_size;
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Connection Refused
|
||||
|
||||
```bash
|
||||
# Check PostgreSQL status
|
||||
sudo systemctl status postgresql
|
||||
|
||||
# Start PostgreSQL
|
||||
sudo systemctl start postgresql
|
||||
|
||||
# Enable on boot
|
||||
sudo systemctl enable postgresql
|
||||
```
|
||||
|
||||
### Permission Denied
|
||||
|
||||
```bash
|
||||
# Grant permissions
|
||||
sudo -u postgres psql -d giglez -c "GRANT ALL ON SCHEMA public TO giglez_user;"
|
||||
sudo -u postgres psql -d giglez -c "GRANT ALL ON ALL TABLES IN SCHEMA public TO giglez_user;"
|
||||
sudo -u postgres psql -d giglez -c "GRANT ALL ON ALL SEQUENCES IN SCHEMA public TO giglez_user;"
|
||||
```
|
||||
|
||||
### PostGIS Not Found
|
||||
|
||||
```bash
|
||||
# Install PostGIS extension
|
||||
sudo apt install postgresql-16-postgis-3
|
||||
|
||||
# Enable in database
|
||||
psql -U giglez_user -d giglez -h localhost -c "CREATE EXTENSION postgis;"
|
||||
```
|
||||
|
||||
### Test PostGIS
|
||||
|
||||
```sql
|
||||
SELECT PostGIS_Version();
|
||||
SELECT ST_AsText(ST_MakePoint(-74.0060, 40.7128));
|
||||
```
|
||||
|
||||
## Architecture Decisions
|
||||
|
||||
Based on Wigle.net analysis (see `docs/wigle_analysis.md`):
|
||||
|
||||
1. **SHA256 primary key**: Automatic deduplication of .sub files
|
||||
2. **PostGIS geometry**: Efficient geospatial queries (GIST indexes)
|
||||
3. **3-table design**: captures → capture_matches → devices
|
||||
4. **Upload markers**: Incremental sync for resumable uploads
|
||||
5. **Materialized views**: Pre-computed statistics for performance
|
||||
6. **Batch transactions**: 512 operations per commit
|
||||
7. **Connection pooling**: 10 base + 20 overflow connections
|
||||
|
||||
## Next Steps
|
||||
|
||||
After database setup:
|
||||
|
||||
1. ✅ Implement SQLAlchemy ORM models (`src/database/models.py`)
|
||||
2. ✅ Create GPS validator module (`src/gps/validator.py`)
|
||||
3. ✅ Set up Alembic migrations (`alembic/`)
|
||||
4. ✅ Build FastAPI upload endpoints (`src/api/`)
|
||||
5. ✅ Import signature databases (Flipper Zero, RTL_433)
|
||||
|
||||
See `IMPLEMENTATION_PLAN.md` for complete roadmap.
|
||||
|
||||
## Resources
|
||||
|
||||
- **Schema Documentation**: `docs/database_schema.md`
|
||||
- **Wigle Analysis**: `docs/wigle_analysis.md`
|
||||
- **Architecture Decisions**: `docs/architecture_decisions.md`
|
||||
- **PostGIS Manual**: https://postgis.net/docs/
|
||||
- **PostgreSQL Documentation**: https://www.postgresql.org/docs/
|
||||
|
||||
---
|
||||
|
||||
**Created**: 2026-01-12
|
||||
**Database Version**: PostgreSQL 16 + PostGIS 3.4
|
||||
**Schema Version**: 1.0.0
|
||||
Reference in New Issue
Block a user