feat: Add wardriving data import system and cleanup tools
Major Features: - Multi-format GPS data import (Wigle CSV, GPS JSON, filename GPS) - Data source tracking (test/mock/production) - Database cleanup tools for managing test data - JSON file persistence for simplified server - UI improvements for map controls Backend: - Added wardriving_importer.py with WigleCSVImporter, GPSJSONImporter, BatchImporter - Added data_source and session_id tracking to captures - New API endpoints: DELETE /api/v1/admin/cleanup - Auto-save/load functionality for captures_simple.json - Updated stats endpoint to show data source breakdown CLI Tools: - scripts/import_wardriving_data.py - Batch import with --data-source flag - scripts/cleanup_database.py - Clean by source, session, or all - scripts/create_test_dataset.py - Generate GPS-tagged test data - scripts/test_wardriving_import.py - Test suite for importers Frontend: - Fixed map controls z-index and positioning issues - Moved controls to top-right to avoid zoom button overlap - Fixed Leaflet zoom controls rendering over header - Changed controls to position:fixed for persistent visibility - Export map object to window.map for proper invalidateSize Documentation: - docs/WARDRIVING_IMPORT.md - Complete import guide - docs/DATA_CLEANUP_GUIDE.md - Cleanup system documentation - docs/TEST_LOCATIONS.md - Test GPS coordinates reference - TEST_RESULTS_SUMMARY.md - Format testing results Test Data: - Created test_dataset_gps with 15 captures across 5 US cities - All test data marked with data_source="test" for easy cleanup Testing: - Verified Wigle CSV import (1 capture from West LA) - Verified GPS filename import (3 captures from UCLA area) - Verified GPS JSON companion files (15 captures, 5 cities) - Verified cleanup functionality (deleted 15 test captures) - Verified data persistence across server restarts 🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,279 @@
|
||||
# Test GPS Locations
|
||||
|
||||
## Overview
|
||||
|
||||
All test GPS coordinates are in **Los Angeles, California** area. This document shows where each test file's GPS coordinates are located.
|
||||
|
||||
## Test Data Locations
|
||||
|
||||
### 📍 Location 1: West Los Angeles
|
||||
**Source**: Wigle CSV Import (`test_export_wigle.csv`)
|
||||
|
||||
```
|
||||
Filename: raw10.sub
|
||||
Latitude: 34.073625
|
||||
Longitude: -118.359557
|
||||
Frequency: 433.92 MHz
|
||||
Area: West LA, near Beverly Hills
|
||||
```
|
||||
|
||||
**Google Maps**: [View Location](https://www.google.com/maps?q=34.073625,-118.359557)
|
||||
|
||||
**Nearby Landmarks**:
|
||||
- West of UCLA campus
|
||||
- Near Beverly Hills
|
||||
- Residential area
|
||||
|
||||
---
|
||||
|
||||
### 📍 Location 2: Near UCLA
|
||||
**Source**: Filename-based GPS
|
||||
|
||||
```
|
||||
Files:
|
||||
- 34.0478N_118.2348W_1637_raw_8.sub (315 MHz)
|
||||
- 34.0478N_118.2349W_1351_raw_10.sub (433.92 MHz)
|
||||
- 34.0478N_118.2349W_1650_test_raw.sub (433.92 MHz)
|
||||
|
||||
Latitude: 34.0478
|
||||
Longitude: -118.2348 / -118.2349
|
||||
Area: Westwood, near UCLA
|
||||
```
|
||||
|
||||
**Google Maps**: [View Location](https://www.google.com/maps?q=34.0478,-118.2348)
|
||||
|
||||
**Nearby Landmarks**:
|
||||
- UCLA campus area
|
||||
- Westwood Village
|
||||
- Education/research district
|
||||
|
||||
---
|
||||
|
||||
### 📍 Location 3: Downtown LA
|
||||
**Source**: GPS JSON files
|
||||
|
||||
```
|
||||
Files: gps_coordinates_*.json
|
||||
Latitude: 34.0522
|
||||
Longitude: -118.2437
|
||||
Area: Downtown Los Angeles
|
||||
```
|
||||
|
||||
**Google Maps**: [View Location](https://www.google.com/maps?q=34.0522,-118.2437)
|
||||
|
||||
**Nearby Landmarks**:
|
||||
- Downtown LA
|
||||
- City Hall area
|
||||
- Financial district
|
||||
|
||||
---
|
||||
|
||||
## Map View
|
||||
|
||||
**All Locations on One Map**: [View All](https://www.google.com/maps/@34.0522,-118.2437,12z)
|
||||
|
||||
```
|
||||
West LA UCLA Area Downtown LA
|
||||
📍 📍 📍
|
||||
(34.0736, -118.3596) (34.0478, -118.2348) (34.0522, -118.2437)
|
||||
| | |
|
||||
|<---- ~7 miles --->|<-- 2 miles -->|
|
||||
| |
|
||||
|<------------ Los Angeles -------->|
|
||||
```
|
||||
|
||||
## Testing Upload
|
||||
|
||||
### Test 1: Upload via Web Interface
|
||||
|
||||
1. Go to http://localhost:8000
|
||||
2. Click "Upload" tab
|
||||
3. Drop file: `34.0478N_118.2349W_1650_test_raw.sub`
|
||||
4. GPS auto-fills: **34.0478, -118.2349**
|
||||
5. Click "Upload Files"
|
||||
6. Go to "Map" tab
|
||||
7. **Should see marker near UCLA**
|
||||
|
||||
### Test 2: Import via CLI
|
||||
|
||||
```bash
|
||||
# Import Wigle CSV (West LA location)
|
||||
./scripts/import_wardriving_data.py signatures/t-embed-rf/test_export_wigle.csv
|
||||
|
||||
# Verify on map
|
||||
curl http://localhost:8000/api/v1/query/captures | jq '.captures[] | {filename, latitude, longitude}'
|
||||
```
|
||||
|
||||
### Test 3: Batch Directory Import
|
||||
|
||||
```bash
|
||||
# Import all GPS-enabled files
|
||||
./scripts/import_wardriving_data.py signatures/t-embed-rf/
|
||||
|
||||
# Check total
|
||||
curl http://localhost:8000/api/v1/stats/summary | jq '.total_captures'
|
||||
```
|
||||
|
||||
## Expected Results
|
||||
|
||||
After uploading all test files, you should see:
|
||||
|
||||
### On Map Tab
|
||||
- **3-4 markers** in Los Angeles area
|
||||
- **Green markers** (315 MHz)
|
||||
- **Blue markers** (433.92 MHz)
|
||||
- Clustered in West LA / UCLA / Downtown areas
|
||||
|
||||
### In Statistics Tab
|
||||
```
|
||||
Total Captures: 4
|
||||
Unique Devices: 2
|
||||
Frequency Distribution:
|
||||
- 315 MHz: 1 capture
|
||||
- 433.92 MHz: 3 captures
|
||||
```
|
||||
|
||||
### In Database Query
|
||||
```json
|
||||
{
|
||||
"total": 4,
|
||||
"captures": [
|
||||
{
|
||||
"filename": "raw10.sub",
|
||||
"latitude": 34.073625,
|
||||
"longitude": -118.359557,
|
||||
"frequency": 433920000
|
||||
},
|
||||
{
|
||||
"filename": "34.0478N_118.2348W_1637_raw_8.sub",
|
||||
"latitude": 34.0478,
|
||||
"longitude": -118.2348,
|
||||
"frequency": 315000000
|
||||
},
|
||||
...
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Verification Commands
|
||||
|
||||
### Check total uploads
|
||||
```bash
|
||||
curl -s http://localhost:8000/api/v1/query/captures | jq '.total'
|
||||
```
|
||||
|
||||
### List all GPS coordinates
|
||||
```bash
|
||||
curl -s http://localhost:8000/api/v1/query/captures | \
|
||||
jq '.captures[] | {filename, lat: .latitude, lon: .longitude}'
|
||||
```
|
||||
|
||||
### View on Google Maps
|
||||
```bash
|
||||
# Get first capture's GPS and open in browser
|
||||
LAT=$(curl -s http://localhost:8000/api/v1/query/captures | jq -r '.captures[0].latitude')
|
||||
LON=$(curl -s http://localhost:8000/api/v1/query/captures | jq -r '.captures[0].longitude')
|
||||
echo "https://www.google.com/maps?q=$LAT,$LON"
|
||||
```
|
||||
|
||||
## Common GPS Formats Reference
|
||||
|
||||
### Format 1: N/S/E/W (Flipper Zero, T-Embed)
|
||||
```
|
||||
34.0478N_118.2349W_filename.sub
|
||||
└─┬─┘└┬┘ └──┬──┘└┬┘
|
||||
│ │ │ └─ West (negative longitude)
|
||||
│ │ └────── Longitude value
|
||||
│ └──────────── North (positive latitude)
|
||||
└──────────────── Latitude value
|
||||
```
|
||||
|
||||
### Format 2: Signed Decimal (RTL-SDR, Custom)
|
||||
```
|
||||
34.0478_-118.2349_filename.sub
|
||||
└─┬─┘ └───┬───┘
|
||||
│ └─────── Negative = West
|
||||
└──────────────── Positive = North
|
||||
```
|
||||
|
||||
### Format 3: lat/lon Prefix
|
||||
```
|
||||
lat34.0478lon-118.2349_filename.sub
|
||||
└──┬──┘ └───┬───┘
|
||||
│ └─────── Longitude
|
||||
└────────────────── Latitude
|
||||
```
|
||||
|
||||
### Format 4: Wigle CSV
|
||||
```csv
|
||||
SignalHash,Frequency(MHz),Protocol,Latitude,Longitude,...
|
||||
hash123,433.920,RAW,34.0478,-118.2349,...
|
||||
└─┬─┘ └───┬───┘
|
||||
│ └─────── Longitude
|
||||
└──────────────── Latitude
|
||||
```
|
||||
|
||||
### Format 5: GPS JSON
|
||||
```json
|
||||
{
|
||||
"data": {
|
||||
"latitude": 34.0478,
|
||||
"longitude": -118.2349,
|
||||
"accuracy": 5.0
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## GPS Coordinate Ranges
|
||||
|
||||
### Valid Ranges
|
||||
- **Latitude**: -90 to 90 (South to North)
|
||||
- Negative = Southern hemisphere
|
||||
- Positive = Northern hemisphere
|
||||
- 0 = Equator
|
||||
|
||||
- **Longitude**: -180 to 180 (West to East)
|
||||
- Negative = Western hemisphere
|
||||
- Positive = Eastern hemisphere
|
||||
- 0 = Prime Meridian (Greenwich)
|
||||
|
||||
### Los Angeles Reference
|
||||
```
|
||||
Los Angeles, CA:
|
||||
Latitude: ~34° N (positive)
|
||||
Longitude: ~118° W (negative)
|
||||
Full coords: 34.0522, -118.2437
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Issue: Markers don't appear on map
|
||||
|
||||
**Check**:
|
||||
1. Verify uploads succeeded: `curl http://localhost:8000/api/v1/query/captures`
|
||||
2. Check GPS coordinates are valid (within range)
|
||||
3. Refresh map page
|
||||
4. Check browser console for errors
|
||||
|
||||
### Issue: Wrong location
|
||||
|
||||
**Check**:
|
||||
1. Verify N/S/E/W in filename (W = negative lon)
|
||||
2. Check coordinate order (lat first, lon second)
|
||||
3. Validate coordinate ranges
|
||||
|
||||
### Issue: Can't see captures on map
|
||||
|
||||
**Solution**:
|
||||
1. Zoom out to see Los Angeles area
|
||||
2. Look for marker clusters (multiple captures at same location)
|
||||
3. Click Statistics tab to verify uploads
|
||||
|
||||
## Summary
|
||||
|
||||
**All test data uses Los Angeles, California coordinates:**
|
||||
- 📍 West LA: 34.073625, -118.359557
|
||||
- 📍 UCLA Area: 34.0478, -118.2348/9
|
||||
- 📍 Downtown: 34.0522, -118.2437
|
||||
|
||||
**View map**: http://localhost:8000 and zoom to Los Angeles to see all test captures! 🗺️
|
||||
Reference in New Issue
Block a user