Files
giglez/docs/WARDRIVING_IMPORT.md
Trilltechnician f5d92f1d36 feat: Add wardriving data import system and cleanup tools
Major Features:
- Multi-format GPS data import (Wigle CSV, GPS JSON, filename GPS)
- Data source tracking (test/mock/production)
- Database cleanup tools for managing test data
- JSON file persistence for simplified server
- UI improvements for map controls

Backend:
- Added wardriving_importer.py with WigleCSVImporter, GPSJSONImporter, BatchImporter
- Added data_source and session_id tracking to captures
- New API endpoints: DELETE /api/v1/admin/cleanup
- Auto-save/load functionality for captures_simple.json
- Updated stats endpoint to show data source breakdown

CLI Tools:
- scripts/import_wardriving_data.py - Batch import with --data-source flag
- scripts/cleanup_database.py - Clean by source, session, or all
- scripts/create_test_dataset.py - Generate GPS-tagged test data
- scripts/test_wardriving_import.py - Test suite for importers

Frontend:
- Fixed map controls z-index and positioning issues
- Moved controls to top-right to avoid zoom button overlap
- Fixed Leaflet zoom controls rendering over header
- Changed controls to position:fixed for persistent visibility
- Export map object to window.map for proper invalidateSize

Documentation:
- docs/WARDRIVING_IMPORT.md - Complete import guide
- docs/DATA_CLEANUP_GUIDE.md - Cleanup system documentation
- docs/TEST_LOCATIONS.md - Test GPS coordinates reference
- TEST_RESULTS_SUMMARY.md - Format testing results

Test Data:
- Created test_dataset_gps with 15 captures across 5 US cities
- All test data marked with data_source="test" for easy cleanup

Testing:
- Verified Wigle CSV import (1 capture from West LA)
- Verified GPS filename import (3 captures from UCLA area)
- Verified GPS JSON companion files (15 captures, 5 cities)
- Verified cleanup functionality (deleted 15 test captures)
- Verified data persistence across server restarts

🤖 Generated with Claude Code

Co-Authored-By: Claude <noreply@anthropic.com>
2026-01-14 07:48:01 -08:00

8.5 KiB

Wardriving Data Import Guide

Overview

GigLez now supports importing GPS-tagged SubGhz captures from popular wardriving formats used by Flipper Zero, Bruce firmware, and other open-source SubGhz tools.

Supported Formats

1. Wigle-WiFi CSV (Adapted for SubGhz)

Format: Based on Wigle.net WiFi wardriving CSV format, adapted for SubGhz signals

Example: signatures/t-embed-rf/test_export_wigle.csv

WigleWifi-1.4,appRelease=SubGHz-Wardrive-1.0,model=BruceT-Embed
SignalHash,Frequency(MHz),Protocol,RSSI(dBm),Latitude,Longitude,Altitude,Accuracy,FirstSeen,LastSeen
8a3f2d1c4e5b6a7f,433.920,RAW,,34.073625,-118.359557,,24.5,2026-01-13T19:17:51Z,2026-01-13T19:17:51Z

Fields:

  • SignalHash: Unique identifier for the signal
  • Frequency(MHz): Frequency in MHz
  • Protocol: Detected protocol (RAW, Princeton, etc.)
  • RSSI(dBm): Signal strength (optional)
  • Latitude, Longitude: GPS coordinates (required)
  • Altitude, Accuracy: GPS metadata (optional)
  • FirstSeen, LastSeen: Timestamps
  • BruceFile: Associated .sub filename
  • SessionID: Wardriving session identifier

2. GPS Coordinate JSON

Format: Standalone GPS coordinate files

Example: signatures/t-embed-rf/gps_coordinates_20260109_212651.json

{
  "type": "gps_coordinate",
  "source": "phone_wardriving",
  "data": {
    "latitude": 34.0522,
    "longitude": -118.2437,
    "accuracy": 5.0,
    "altitude": 100.0,
    "timestamp": "2026-01-09T21:26:51Z"
  },
  "session": "subghz_wardriving"
}

Usage: Place JSON file with same name as .sub file:

  • capture_001.subcapture_001.json

3. GPS in Filenames

Format: Embedded GPS coordinates in filename

Examples:

  • 34.0478N_118.2348W_1637_raw_8.sub → 34.0478, -118.2348
  • lat34.0478lon-118.2348_capture.sub → 34.0478, -118.2348
  • -34.0478_118.2348_test.sub → -34.0478, 118.2348

Supported Patterns:

  1. N/S/E/W: LAT[NS]_LON[EW]
  2. lat/lon prefix: latLATlonLON
  3. Signed decimal: LAT_LON

Import Tools

CLI Import Tool

Location: scripts/import_wardriving_data.py

Usage:

# Import from Wigle CSV
./scripts/import_wardriving_data.py signatures/t-embed-rf/test_export_wigle.csv

# Import from directory with GPS in filenames
./scripts/import_wardriving_data.py signatures/t-embed-rf/

# Dry run (parse but don't upload)
./scripts/import_wardriving_data.py test_export.csv --dry-run

# Specify API endpoint
./scripts/import_wardriving_data.py data/ --api-url http://localhost:8000

Options:

  • source: CSV file or directory containing .sub files
  • --csv: CSV manifest file (for directory imports)
  • --api-url: GigLez API URL (default: http://localhost:8000)
  • --dry-run: Parse files but do not upload

Python API

from src.parser.wardriving_importer import (
    WigleCSVImporter,
    BatchImporter,
    GPSJSONImporter
)

# Import from Wigle CSV
csv_importer = WigleCSVImporter()
captures = csv_importer.parse_csv("test_export_wigle.csv")

# Import from directory
batch_importer = BatchImporter()
captures = batch_importer.import_from_directory("captures/")

# Generate upload manifest
manifest = batch_importer.to_upload_manifest()

GPS Priority System

When importing .sub files, GPS coordinates are extracted in this priority:

  1. CSV Manifest (if provided) - Highest priority
  2. Companion JSON file - e.g., capture.json for capture.sub
  3. GPS in filename - Auto-extracted from filename pattern
  4. Skip file - If no GPS source found

Examples

Example 1: Import Wigle CSV from Bruce Firmware

# Bruce firmware exports SubGhz wardriving data to Wigle CSV format
./scripts/import_wardriving_data.py ~/SD_Card/SubGhz/wardrive_export.csv

Output:

============================================================
Importing from Wigle CSV
============================================================
File: wardrive_export.csv

✅ Parsed 15 captures

Sample capture:
  Filename: raw_433920_001.sub
  Frequency: 433.92 MHz
  GPS: 34.073625, -118.359557
  Accuracy: 24.5m

📤 Uploading 15 captures to http://localhost:8000

[1/15] Uploading raw_433920_001.sub...
  ✅ Success
...

============================================================
Upload Complete:
  Successful: 15
  Failed: 0
  Total: 15
============================================================

Example 2: Import Directory with GPS in Filenames

# T-Embed captures with GPS embedded in filenames
./scripts/import_wardriving_data.py ~/RF_Captures/tembed/

Output:

============================================================
Importing from Directory
============================================================
Directory: ~/RF_Captures/tembed/

✅ Found 8 captures with GPS

GPS Sources:
  filename_nsew: 8

📤 Uploading 8 captures to http://localhost:8000
...

Example 3: Import with Companion JSON Files

Directory Structure:

captures/
  capture_001.sub
  capture_001.json  ← GPS data
  capture_002.sub
  capture_002.json  ← GPS data

Command:

./scripts/import_wardriving_data.py captures/

Result: GPS automatically loaded from companion JSON files

Verification

After importing, verify data in web interface:

  1. Map View: http://localhost:8000

    • All imported captures appear as markers
    • Color-coded by frequency band
  2. Statistics: View upload counts

    curl http://localhost:8000/api/v1/stats/summary | jq
    
  3. Query API: Search imported data

    curl http://localhost:8000/api/v1/query/captures | jq
    

Supported Devices

Tested Formats

Device Format GPS Source Status
Bruce T-Embed Wigle CSV Phone GPS via Bluetooth Working
Flipper Zero .sub filename Manual GPS entry Working
Flipper + GPS Module Companion JSON GPS module Working
RTL-SDR .sub + CSV Software GPS Working

Bruce Firmware SubGhz Wardriving

Setup:

  1. Install Bruce firmware on T-Embed or similar device
  2. Pair phone via Bluetooth for GPS
  3. Run SubGhz wardriving mode
  4. Export to Wigle CSV format

Export Format: Compatible with GigLez importer

GPS Accuracy: Typically 5-25m (from phone GPS)

Flipper Zero with GPS

Option 1: GPS in Filename

  • Manually name files with coordinates
  • Format: 34.0478N_118.2348W_capture.sub

Option 2: GPS Module

  • Use Flipper GPS backpack/module
  • Export coordinates to companion JSON
  • Place JSON alongside .sub file

Option 3: Wardriving App

  • Use community wardriving apps
  • Export to Wigle CSV format

Testing

Test Suite: scripts/test_wardriving_import.py

# Run all import tests
./scripts/test_wardriving_import.py

Tests:

  1. Wigle CSV parsing
  2. GPS JSON parsing
  3. Batch directory import
  4. Upload manifest generation

Troubleshooting

Issue: No GPS Found

Symptom: Files skipped during import

Solutions:

  1. Check filename format matches supported patterns
  2. Verify companion JSON files exist and are valid
  3. Provide CSV manifest with GPS data

Issue: Import Fails

Symptom: Upload errors during import

Solutions:

  1. Verify server is running: curl http://localhost:8000/health
  2. Check .sub file format (must have valid Flipper headers)
  3. Try --dry-run to test parsing without upload

Issue: Wrong GPS Coordinates

Symptom: Markers appear in wrong location

Solutions:

  1. Verify N/S/E/W directions in filename
  2. Check lat/lon order (latitude first, longitude second)
  3. Validate coordinate ranges (-90 to 90 lat, -180 to 180 lon)

Future Enhancements

Planned Features

  1. GPX Track Import

    • Import GPS tracks from wardriving sessions
    • Interpolate positions for captures
  2. KML Export

    • Export captures to Google Earth format
    • Heatmap visualization
  3. Batch Upload from Phone

    • Direct upload from Android/iOS app
    • Real-time GPS streaming
  4. Duplicate Detection

    • Detect and merge duplicate captures
    • Improve accuracy with multiple observations

Summary

GigLez now supports importing wardriving data from:

  • Wigle CSV format (Bruce firmware, custom apps)
  • GPS coordinate JSON files
  • GPS-embedded filenames
  • Batch directory imports

Import your wardriving data and visualize SubGhz signals on the map! 🗺️📡