GPS Auto-Extraction + First Successful Upload Complete

Major milestone: GPS coordinates now auto-extract from filenames and
uploads appear on map with full end-to-end workflow functional!

 GPS Auto-Extraction Features:
- JavaScript GPS extractor class matching Python patterns
- Supports 3 filename formats:
  * N/S/E/W: 34.0478N_118.2349W_filename.sub
  * lat/lon prefix: lat34.0478lon-118.2348_filename.sub
  * Signed decimal: -34.0478_118.2348_filename.sub
- Auto-populates latitude/longitude form fields on file drop
- Green notification toast shows detected coordinates
- File list shows GPS badge for files with coordinates

🗺️ Web Interface Improvements:
- Upload endpoint now stores captures in-memory
- Query endpoint returns uploaded captures for map display
- Stats endpoint shows real-time upload counts
- Map displays uploaded captures as markers
- Color-coded by frequency band

📁 Updated Files:
- static/js/upload.js: GPS extraction + auto-population
- src/api/main_simple.py: In-memory storage + endpoints
- src/parser/gps_extractor.py: Backend GPS extraction (Python)
- scripts/test_gps_extraction.py: Python test suite
- test_gps_extraction.html: Browser test suite

📊 T-Embed Files Updated:
- 34.0478N_118.2348W_1637_raw_8.sub: 315 MHz Princeton
- 34.0478N_118.2349W_1351_raw_10.sub: 433.92 MHz Princeton
- 34.0478N_118.2349W_1650_test_raw.sub: 433.92 MHz RAW
- All now have proper Flipper SubGhz headers

 Tested Features:
- GPS extraction from filename: 34.0478N_118.2349W → 34.0478, -118.2349
- Auto-population of GPS fields in upload form
- File upload with GPS validation
- Capture appears on map after upload
- Statistics update in real-time
- Frequency distribution calculated correctly

🎯 End-to-End Flow Working:
1. User drops .sub file with GPS in filename
2. GPS auto-detected and form fields populate
3. User clicks Upload
4. Server parses RF data + GPS coordinates
5. Capture stored in memory
6. Map refreshes and displays new marker
7. Stats update with new counts

🚀 Demo: http://localhost:8000
Upload 34.0478N_118.2349W_1351_raw_10.sub and watch it appear on map!

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
2026-01-13 18:37:13 -08:00
parent 48fcb00241
commit 04bd80b25b
33 changed files with 1903 additions and 22 deletions
+136
View File
@@ -0,0 +1,136 @@
#!/bin/bash
# Create giglez repository on Gitea and push
# Based on gitea-command-center pattern
set -e
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
RED='\033[0;31m'
BLUE='\033[0;34m'
NC='\033[0m'
echo -e "${BLUE}🛰️ Creating GigLez Repository on Gitea${NC}"
echo "========================================="
echo ""
# Get token from gopass
echo "Retrieving API token from gopass..."
TOKEN=$(gopass show -o gitea/api-tokens/tea-cli 2>/dev/null)
if [ -z "$TOKEN" ]; then
echo -e "${RED}❌ Token not found in gopass${NC}"
echo "Please setup gopass token first"
exit 1
fi
echo -e "${GREEN}✅ Token retrieved${NC}"
# Get username from API
echo "Getting username..."
USER_INFO=$(curl -sk -H "Authorization: token $TOKEN" https://localhost:3030/api/v1/user)
USERNAME=$(echo "$USER_INFO" | jq -r '.login')
if [ -z "$USERNAME" ] || [ "$USERNAME" = "null" ]; then
echo -e "${RED}❌ Could not get username from API${NC}"
echo "Is Gitea running on localhost:3030?"
echo "Response: $USER_INFO"
exit 1
fi
echo -e "${GREEN}✅ Logged in as: $USERNAME${NC}"
echo ""
# Check if repository exists
echo "Checking if giglez repository exists..."
REPO_CHECK=$(curl -sk -H "Authorization: token $TOKEN" \
https://localhost:3030/api/v1/repos/$USERNAME/giglez)
if echo "$REPO_CHECK" | jq -e '.id' > /dev/null 2>&1; then
echo -e "${YELLOW}⚠️ Repository already exists${NC}"
REPO_EXISTS=true
else
echo -e "${GREEN}✅ Repository doesn't exist, will create${NC}"
REPO_EXISTS=false
fi
# Create repository if it doesn't exist
if [ "$REPO_EXISTS" = false ]; then
echo ""
echo "Creating giglez repository..."
CREATE_RESPONSE=$(curl -sk -X POST \
-H "Authorization: token $TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "giglez",
"description": "IoT RF Device Mapping Platform - Wigle for Sub-GHz Signals",
"private": false,
"default_branch": "main"
}' \
https://localhost:3030/api/v1/user/repos)
if echo "$CREATE_RESPONSE" | jq -e '.id' > /dev/null 2>&1; then
echo -e "${GREEN}✅ Repository created successfully${NC}"
else
echo -e "${RED}❌ Failed to create repository${NC}"
echo "$CREATE_RESPONSE" | jq .
exit 1
fi
fi
echo ""
echo -e "${BLUE}📡 Setting up Git remote${NC}"
echo "================================"
# Get Gitea URL (check for Tailscale or local)
GITEA_URL="https://localhost:3030"
REMOTE_URL="${GITEA_URL}/${USERNAME}/giglez.git"
# Check if origin already exists
if git remote get-url origin > /dev/null 2>&1; then
echo -e "${YELLOW}⚠️ Remote 'origin' already exists${NC}"
CURRENT_ORIGIN=$(git remote get-url origin)
echo " Current: $CURRENT_ORIGIN"
echo " New: $REMOTE_URL"
read -p "Replace origin? (y/n) " -n 1 -r
echo
if [[ $REPLY =~ ^[Yy]$ ]]; then
git remote remove origin
git remote add origin "$REMOTE_URL"
echo -e "${GREEN}✅ Remote 'origin' updated${NC}"
fi
else
git remote add origin "$REMOTE_URL"
echo -e "${GREEN}✅ Remote 'origin' added${NC}"
fi
echo ""
echo -e "${BLUE}🚀 Pushing to Gitea${NC}"
echo "================================"
# Get current branch
CURRENT_BRANCH=$(git branch --show-current)
echo "Current branch: $CURRENT_BRANCH"
# Push
echo "Pushing $CURRENT_BRANCH to origin..."
if git push -u origin "$CURRENT_BRANCH"; then
echo -e "${GREEN}✅ Successfully pushed to Gitea!${NC}"
else
echo -e "${RED}❌ Push failed${NC}"
exit 1
fi
echo ""
echo "========================================="
echo -e "${GREEN}🎉 Repository Setup Complete!${NC}"
echo "========================================="
echo ""
echo "Repository URL: ${GITEA_URL}/${USERNAME}/giglez"
echo "Git Remote: $REMOTE_URL"
echo "Branch: $CURRENT_BRANCH"
echo ""
echo "View your repository:"
echo " ${GITEA_URL}/${USERNAME}/giglez"
echo ""
+43
View File
@@ -0,0 +1,43 @@
# Gitea Repository Setup
## Creating the Repository
### On Gitea Web Interface
1. **Login to your Gitea instance**
2. **Click "+" → "New Repository"**
3. **Fill in details**:
- Repository Name: `giglez`
- Description: `IoT RF Device Mapping Platform - Wigle for Sub-GHz Signals`
- Visibility: Private (or Public)
- Initialize: ❌ **DO NOT** initialize with README (we already have code)
4. **Click "Create Repository"**
### Add Remote
```bash
# Replace with your Gitea URL
git remote add origin https://your-gitea-server.com/your-username/giglez.git
# Or with SSH
git remote add origin git@your-gitea-server.com:your-username/giglez.git
```
### Push to Gitea
```bash
# Push current branch
git push -u origin p1-p2-validation
# Or push all branches
git push -u origin --all
```
## Current Commit
**Branch**: `p1-p2-validation`
**Commit**: Phase 3 Complete: Web Interface MVP
**Files**: 39 files changed, 10,138 insertions(+)
**Ready to push** when you provide Gitea URL!
+226
View File
@@ -0,0 +1,226 @@
# GPS Auto-Extraction from Filenames
## Feature Overview
The web interface now automatically detects and extracts GPS coordinates from filenames, eliminating the need for manual coordinate entry when filenames contain location data.
## Supported Filename Patterns
### 1. N/S/E/W Format (Primary)
**Pattern**: `LAT[NS]_LON[EW]`
**Examples**:
- `34.0478N_118.2348W_1637_raw_8.sub` → 34.0478, -118.2348
- `34.0478N_118.2349W_1351_raw_10.sub` → 34.0478, -118.2349
- `40.7128N_74.0060W_capture.sub` → 40.7128, -74.0060
**Notes**:
- Most common format from T-Embed RF captures
- N = positive latitude, S = negative latitude
- E = positive longitude, W = negative longitude
### 2. lat/lon Prefix Format
**Pattern**: `latLATlonLON`
**Examples**:
- `lat34.0478lon-118.2348_test.sub` → 34.0478, -118.2348
- `lat40.7128lon-74.0060_capture.sub` → 40.7128, -74.0060
**Notes**:
- Allows signed decimal notation
- Case-insensitive (LAT/LON also works)
### 3. Signed Decimal Format
**Pattern**: `LAT_LON` (with optional negative signs)
**Examples**:
- `-34.0478_118.2348_capture.sub` → -34.0478, 118.2348
- `40.7128_-74.0060_test.sub` → 40.7128, -74.0060
**Notes**:
- Simple signed decimal degrees
- Less common but supported
## How It Works
### 1. File Upload Detection
When user selects/drops .sub files:
```javascript
1. Files are added to upload queue
2. First file with GPS coordinates is automatically detected
3. GPS fields are auto-populated
4. User sees notification toast
```
### 2. UI Indicators
- **GPS Detection Notification**: Green toast shows which file provided coordinates
- **File List Badge**: Files with GPS show 📍 GPS badge
- **Coordinate Display**: Extracted coordinates appear next to filename
### 3. Validation
- Latitude range: -90 to 90
- Longitude range: -180 to 180
- Invalid coordinates are rejected
- Malformed patterns are skipped
## User Experience
### Upload Flow with GPS in Filename
1. **Drop/Select File**: User uploads `34.0478N_118.2348W_capture.sub`
2. **Auto-Detection**: System extracts GPS (34.0478, -118.2348)
3. **Notification**: Green toast appears:
```
📍 GPS Auto-Detected!
From: 34.0478N_118.2348W_capture.sub
Coordinates: 34.047800, -118.234800
```
4. **Form Population**: Latitude and longitude fields auto-fill
5. **Upload**: User clicks "Upload Files" (no manual GPS entry needed)
### Upload Flow WITHOUT GPS in Filename
1. **Drop/Select File**: User uploads `raw_7.sub`
2. **No Detection**: No GPS found in filename
3. **Manual Entry**: User must manually enter GPS or use "Use Current Location"
4. **Validation**: System checks for valid coordinates before upload
## Implementation Details
### JavaScript GPS Extractor Class
Location: `static/js/upload.js`
```javascript
class GPSFilenameExtractor {
extract(filename) {
// Try each pattern in order
return this.tryNSEW(filename) ||
this.tryLatLon(filename) ||
this.trySigned(filename);
}
}
```
### Pattern Matching
- **PATTERN_NSEW**: `/(\d+\.?\d*)([NS])_(\d+\.?\d*)([EW])/i`
- **PATTERN_LATLON**: `/lat(-?\d+\.?\d+)lon(-?\d+\.?\d+)/i`
- **PATTERN_SIGNED**: `/(-?\d+\.?\d+)_(-?\d+\.?\d+)/`
### Auto-Population Logic
```javascript
function addFiles(files) {
// Try to extract GPS from first file with coordinates
if (!autoDetectedGPS) {
for (const file of files) {
const coords = gpsExtractor.extract(file.name);
if (coords) {
autoDetectedGPS = coords;
document.getElementById('default-lat').value = coords.latitude.toFixed(6);
document.getElementById('default-lon').value = coords.longitude.toFixed(6);
showGPSDetectionNotification(file.name, coords);
break;
}
}
}
}
```
## Testing
### Test Suite
Location: `test_gps_extraction.html`
Open in browser to verify all patterns:
```bash
firefox test_gps_extraction.html
# or
chromium test_gps_extraction.html
```
### Test Coverage
- ✅ N/S/E/W format (3 test cases)
- ✅ lat/lon prefix format
- ✅ Signed decimal format
- ✅ Negative detection (files without GPS)
### Python Compatibility
The JavaScript implementation matches the Python `GPSFilenameExtractor` class:
- Same pattern support
- Same validation rules
- Same coordinate transformation logic
## T-Embed RF Compatibility
### Tested Files
From `signatures/t-embed-rf/`:
- ✅ `34.0478N_118.2348W_1637_raw_8.sub` → Auto-detected
- ✅ `34.0478N_118.2349W_1351_raw_10.sub` → Auto-detected
- ✅ `34.0478N_118.2349W_1650_test_raw.sub` → Auto-detected
- ⏭️ `raw_4.sub` → Manual entry required
- ⏭️ `raw_5.sub` → Manual entry required
**Success Rate**: 37.5% (3 of 8 files) auto-detected
## Error Messages
### GPS Required
If no GPS provided (manual or filename):
```
Please provide GPS coordinates (manually or via filename like: 34.0478N_118.2349W_filename.sub)
```
### GPS Detected but Not Populated
If GPS exists in filename but form is empty:
```
GPS detected in filename but not populated. Please refresh and try again.
```
### Out of Range
If coordinates exceed valid ranges:
```
GPS coordinates out of range
```
## Benefits
1. **Faster Uploads**: No manual GPS entry for T-Embed captures
2. **Fewer Errors**: Eliminates coordinate typos
3. **Better UX**: Clear feedback when GPS is detected
4. **Flexible**: Still allows manual entry for files without GPS
5. **Compatible**: Matches Python backend extraction logic
## Future Enhancements
### Additional Patterns
- DMS format: `34d02m52sN_118d14m05sW`
- Compact format: `N34.0478W118.2348`
- Plus codes: `8762+MXP_capture.sub`
### Multi-File GPS
Currently uses first file's GPS for all uploads. Future enhancement:
- Per-file GPS extraction
- Mixed batch uploads (some with GPS, some without)
- GPS override UI per file
### Companion JSON
Check for `.json` files alongside `.sub` files:
```json
{
"capture.sub": {
"latitude": 34.0478,
"longitude": -118.2348,
"accuracy": 5.0
}
}
```
## Related Files
- `static/js/upload.js` - Frontend GPS extraction
- `src/parser/gps_extractor.py` - Backend GPS extraction
- `scripts/test_gps_extraction.py` - Python test suite
- `test_gps_extraction.html` - JavaScript test suite
## Summary
GPS auto-extraction is **LIVE** and working with the format `34.0478N_118.2349W_filename.sub`. Users can now simply drag and drop T-Embed captures without manually entering coordinates!
View File
+92
View File
@@ -0,0 +1,92 @@
#!/usr/bin/env python3
"""
Test GPS extraction from T-Embed files
Demonstrates automatic GPS population from filenames
"""
import sys
from pathlib import Path
# Add project root to path
sys.path.insert(0, str(Path(__file__).parent.parent))
from src.parser.gps_extractor import GPSFilenameExtractor
def main():
"""Test GPS extraction on T-Embed files"""
print("=" * 80)
print("GPS EXTRACTION TEST - T-Embed Files")
print("=" * 80)
print()
# Find T-Embed files
tembed_dir = Path(__file__).parent.parent / 'signatures' / 't-embed-rf'
if not tembed_dir.exists():
print(f"❌ T-Embed directory not found: {tembed_dir}")
return 1
sub_files = list(tembed_dir.glob('*.sub'))
print(f"Found {len(sub_files)} .sub files")
print()
# Test extraction
extractor = GPSFilenameExtractor()
with_gps = []
without_gps = []
for sub_file in sorted(sub_files):
coords = extractor.extract(sub_file.name)
if coords:
with_gps.append((sub_file.name, coords))
print(f"{sub_file.name}")
print(f" 📍 Latitude: {coords.latitude:.6f}")
print(f" 📍 Longitude: {coords.longitude:.6f}")
print(f" 📋 Source: {coords.source}")
else:
without_gps.append(sub_file.name)
print(f"⏭️ {sub_file.name} - No GPS in filename")
print()
# Summary
print("=" * 80)
print("SUMMARY")
print("=" * 80)
print(f"Total files: {len(sub_files)}")
print(f"With GPS coords: {len(with_gps)} ({len(with_gps)/len(sub_files)*100:.1f}%)")
print(f"Without GPS coords: {len(without_gps)} ({len(without_gps)/len(sub_files)*100:.1f}%)")
print()
if with_gps:
print("Files with GPS coordinates:")
for filename, coords in with_gps:
print(f" - {filename}")
print(f" {coords.latitude:.6f}, {coords.longitude:.6f}")
print()
if without_gps:
print("Files without GPS (would need manual entry or JSON companion):")
for filename in without_gps:
print(f" - {filename}")
print()
print("=" * 80)
print("NEXT STEPS")
print("=" * 80)
print("1. Files with GPS in filename → Auto-populate coordinates")
print("2. Files without GPS → Look for companion JSON files")
print("3. If no JSON found → Require manual GPS entry during upload")
print("=" * 80)
return 0
if __name__ == '__main__':
sys.exit(main())
+114
View File
@@ -0,0 +1,114 @@
# T-Embed RF SubGHz Wardriving Export
**Export Date:** 2026-01-13
**Device:** Bruce T-Embed CC1101 (ESP32-S3)
**GPS Source:** Phone (termux-location)
**Format:** Wigle-compatible CSV
## Files in This Export
### `test_export_wigle.csv`
Wigle-compatible CSV format with GPS+RF signal pairs.
**Format:**
- **SignalHash:** SHA256 hash of frequency+protocol (16 chars)
- **Frequency(MHz):** RF frequency in MHz
- **Protocol:** SubGHz protocol (RAW, Princeton, etc.)
- **Latitude/Longitude:** GPS coordinates (decimal degrees)
- **FirstSeen/LastSeen:** ISO 8601 timestamps
- **CaptureCount:** Number of times signal was observed
- **BruceFile:** Original .sub filename from Bruce device
### `raw10.sub`
Flipper Zero format SubGHz capture file from Bruce T-Embed.
**Format:**
```
Filetype: Flipper SubGhz RAW File
Version: 1
Frequency: 433920000 (433.92 MHz)
Preset: FCC_433
Protocol: RAW
RAW_Data: <timing data>
```
### `quick_capture.py`
Python script to capture GPS+RF pairs and store in database.
**Usage:**
```bash
python3 quick_capture.py <bruce_file.sub> [session_id]
```
## Current GPS Location
**Latest Capture:**
- **Coordinates:** 34.073625°N, -118.359557°W
- **Accuracy:** 24.5 meters
- **Timestamp:** 2026-01-13T19:17:51Z
- **Provider:** GPS
**Location:** Greater Los Angeles area, California, USA
## Database Schema
Stored in: `~/logs/subghz_wardriving.db` (on phone)
**Tables:**
- `wardriving_sessions` - Session metadata
- `gps_track` - GPS breadcrumb trail
- `subghz_signals` - RF captures with GPS correlation
## Wigle CSV Sample
```csv
WigleWifi-1.4,appRelease=SubGHz-Wardrive-1.0,model=BruceT-Embed...
SignalHash,Frequency(MHz),Protocol,RSSI(dBm),Latitude,Longitude...
8a3f2d1c4e5b6a7f,433.920,RAW,,34.073625,-118.359557,24.5...
```
## Hardware
**Bruce T-Embed CC1101:**
- **RF Module:** Texas Instruments CC1101
- **Frequency Range:** 300-928 MHz
- **Modulations:** ASK, FSK, GFSK, MSK, 2-FSK, 4-FSK
- **Data Rate:** 0.6 - 500 kBaud
- **RX Sensitivity:** -112 dBm @ 1.2 kBaud
- **TX Power:** -30 to +10 dBm
**Phone GPS:**
- **Providers:** GPS, Network, Passive
- **Typical Accuracy:** 10-50 meters
- **Update Interval:** 158.4 seconds (2.64 minutes) for wardriving
## Next Steps
1. **Analyze Signals:** Plot frequency distribution
2. **Map Visualization:** Create heatmap of signal locations
3. **Protocol Decode:** Identify known protocols (garage doors, car keys, weather stations)
4. **Expand Collection:** Add more capture sessions
5. **Cross-reference:** Compare with FCC database for legitimate transmitters
## Resources
- **Bruce Firmware:** https://github.com/pr3y/Bruce
- **Flipper SubGHz:** https://docs.flipperzero.one/sub-ghz
- **CC1101 Datasheet:** https://www.ti.com/product/CC1101
- **Wigle.net:** https://wigle.net/ (WiFi wardriving reference)
## Security Notes
This system is designed for:
- ✅ Security research and education
- ✅ RF spectrum analysis
- ✅ Authorized penetration testing
- ✅ Personal network security assessment
-**NOT for:** Unauthorized access, signal replay attacks, or malicious use
---
**System:** SubGHz Wigle-Like Wardriving v1.0
**Documentation:** ~/docs/SUBGHZ_WIGLE_WARDRIVING.md
**Database:** ~/logs/subghz_wardriving.db
**Export Tool:** ~/python/subghz_wardriving/exporter.py
@@ -0,0 +1,391 @@
# SubGHz Wigle-Like Wardriving System
**Complete GPS+RF Capture and Export System**
## Overview
This system creates a Wigle-like wardriving platform for sub-GHz RF signals, pairing GPS coordinates with Bruce T-Embed RF captures in a standardized format similar to Wigle WiFi wardriving.
**Architecture:**
```
┌──────────────────────────────────────────────────────────┐
│ GPS Sources (termux-location) │
└─────────────────┬────────────────────────────────────────┘
┌──────────────────────────────────────────────────────────┐
│ Bruce T-Embed RF Captures (.sub files) │
│ - Frequency, Protocol, Preset │
│ - RAW data, Key data │
└─────────────────┬────────────────────────────────────────┘
┌──────────────────────────────────────────────────────────┐
│ SubGHz Wardriving Database (~/logs/subghz_wardriving.db) │
│ ├─ wardriving_sessions │
│ ├─ gps_track (GPS breadcrumbs) │
│ └─ subghz_signals (RF+GPS pairs) │
└─────────────────┬────────────────────────────────────────┘
┌──────────────────────────────────────────────────────────┐
│ Export Formats │
│ ├─ JSON (detailed) │
│ ├─ CSV (standard wardriving) │
│ ├─ Wigle CSV (standardized for sharing) │
│ └─ SQLite (database backup) │
└──────────────────────────────────────────────────────────┘
```
## Quick Start
### 1. Capture GPS + Associate with RF File
```bash
# Capture current GPS and pair with Bruce .sub file
python3 ~/python/subghz_wardriving/quick_capture.py \
~/projects/device-integration/bruce/data/captures/raw10.sub
# With custom session name
python3 ~/python/subghz_wardriving/quick_capture.py \
raw10.sub downtown_survey
```
**Latest GPS:**
- **Coordinates:** 34.07362505, -118.35955673
- **Accuracy:** 24.5m
- **Timestamp:** 2026-01-13T11:17:51-08:00
- **Provider:** GPS
### 2. Export to Wigle Format
```bash
# Export all data to Wigle CSV
gps wigle-export
# Export specific session
gps wigle-export --session downtown_survey
# Direct export
python3 -c "
from subghz_wardriving.exporter import SubGHzExporter, ExportConfig
exporter = SubGHzExporter()
config = ExportConfig(format='wigle', compress=False)
result = exporter.export_session(config)
print(f'Exported to: {result[\"output_file\"]}')
"
```
**Output Location:** `~/exports/subghz_wardriving/`
## Database Schema
### Table: `subghz_signals`
Stores RF captures paired with GPS coordinates:
| Column | Type | Description |
|--------|------|-------------|
| `id` | INTEGER | Primary key |
| `session_id` | TEXT | Wardriving session ID |
| `capture_id` | TEXT | Unique capture identifier |
| `timestamp` | TEXT | ISO 8601 timestamp |
| `bruce_filename` | TEXT | Original .sub filename |
| `bruce_file_path` | TEXT | Full path to .sub file |
| `file_size_bytes` | INTEGER | File size |
| `frequency_hz` | INTEGER | RF frequency in Hz |
| `protocol` | TEXT | Protocol (RAW, Princeton, etc.) |
| `gps_latitude` | REAL | Latitude (decimal degrees) |
| `gps_longitude` | REAL | Longitude (decimal degrees) |
| `gps_accuracy_meters` | REAL | GPS accuracy |
| `interpolated` | BOOLEAN | GPS interpolated? |
| `interpolation_confidence` | REAL | Confidence score (0-1) |
| `time_gap_seconds` | REAL | Time since last capture |
| `distance_traveled_meters` | REAL | Distance from last capture |
| `estimated_speed_mps` | REAL | Speed in m/s |
| `file_hash_md5` | TEXT | MD5 hash of file |
| `notes` | TEXT | Additional notes |
### Table: `gps_track`
GPS breadcrumb trail:
| Column | Type | Description |
|--------|------|-------------|
| `id` | INTEGER | Primary key |
| `session_id` | TEXT | Session identifier |
| `timestamp` | TEXT | ISO 8601 timestamp |
| `latitude` | REAL | Latitude |
| `longitude` | REAL | Longitude |
| `altitude` | REAL | Altitude (meters) |
| `accuracy` | REAL | Horizontal accuracy (meters) |
| `speed` | REAL | Speed (m/s) |
| `provider` | TEXT | GPS provider (gps/network/passive) |
### Table: `wardriving_sessions`
Session metadata:
| Column | Type | Description |
|--------|------|-------------|
| `session_id` | TEXT | Primary key |
| `start_time` | TEXT | Session start time |
| `end_time` | TEXT | Session end time |
| `total_signals` | INTEGER | Number of signals captured |
| `total_distance_km` | REAL | Total distance traveled |
| `device_info` | TEXT | Device information |
| `notes` | TEXT | Session notes |
## Wigle CSV Format
### Header Format
```csv
WigleWifi-1.4,appRelease=SubGHz-Wardrive-1.0,model=BruceT-Embed,release=Android,device=Phone,display=SubGHzWardrive,board=termux,brand=custom
SignalHash,Frequency(MHz),Protocol,RSSI(dBm),Latitude,Longitude,Altitude,Accuracy,FirstSeen,LastSeen,CaptureCount,Interpolated,Confidence,BruceFile,SessionID
```
### Columns
| Column | Description |
|--------|-------------|
| `SignalHash` | SHA256 hash of frequency+protocol (16 chars) |
| `Frequency(MHz)` | Frequency in MHz (e.g., 433.92) |
| `Protocol` | Protocol name (RAW, Princeton, etc.) |
| `RSSI(dBm)` | Signal strength (not available from .sub files) |
| `Latitude` | Decimal degrees |
| `Longitude` | Decimal degrees |
| `Altitude` | Meters above sea level |
| `Accuracy` | GPS accuracy in meters |
| `FirstSeen` | ISO 8601 timestamp of first capture |
| `LastSeen` | ISO 8601 timestamp of last capture |
| `CaptureCount` | Number of times signal was seen |
| `Interpolated` | true/false - was GPS interpolated? |
| `Confidence` | Interpolation confidence (0-1) |
| `BruceFile` | Original .sub filename |
| `SessionID` | Wardriving session ID |
### Signal Aggregation
Signals are grouped by hash (frequency + protocol), similar to how Wigle groups WiFi by BSSID. Multiple captures of the same signal show:
- First and last seen timestamps
- Total capture count
- Location from first capture
## Usage Examples
### Example 1: Quick Capture
```bash
# Capture GPS now with Bruce file
python3 ~/python/subghz_wardriving/quick_capture.py raw10.sub
```
**Output:**
```
📡 SubGHz GPS Capture
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Bruce File: raw10.sub
Session: quick_20260113_112045
🛰️ Getting GPS location...
✓ GPS Location: 34.073625, -118.359557
Accuracy: 24.5m
📻 Parsing RF file...
✓ Frequency: 433920000 Hz (433.920 MHz)
Protocol: RAW
Preset: FCC_433
💾 Storing to database...
✓ Stored signal: quick_20260113_112045_raw10_1736789645
✅ Capture paired successfully!
Database: ~/logs/subghz_wardriving.db
```
### Example 2: Export Session
```bash
# Export to Wigle CSV
gps wigle-export --session downtown_survey
```
**Output File:** `~/exports/subghz_wardriving/downtown_survey_20260113_112130_wigle.csv`
**Content:**
```csv
WigleWifi-1.4,appRelease=SubGHz-Wardrive-1.0,model=BruceT-Embed,release=Android,device=Phone,display=SubGHzWardrive,board=termux,brand=custom
SignalHash,Frequency(MHz),Protocol,RSSI(dBm),Latitude,Longitude,Altitude,Accuracy,FirstSeen,LastSeen,CaptureCount,Interpolated,Confidence,BruceFile,SessionID
8a3f2d1c4e5b6a7f,433.920,RAW,,34.073625,-118.359557,,24.5,2026-01-13T19:17:51Z,2026-01-13T19:17:51Z,1,false,1.0,raw10.sub,downtown_survey
```
### Example 3: Wardriving Session
```bash
# Start GPS logging
gps on --session wardrive_session1 --interval 158.4
# Bruce captures RF signals to /BruceRF/*.sub files
# (Manually on Bruce device)
# After session, run batch import
for subfile in ~/projects/device-integration/bruce/data/captures/*.sub; do
python3 ~/python/subghz_wardriving/quick_capture.py "$subfile" wardrive_session1
done
# Stop GPS logging
gps off
# Export to Wigle format
gps wigle-export --session wardrive_session1
```
## File Locations
| Component | Path |
|-----------|------|
| **Database** | `~/logs/subghz_wardriving.db` |
| **GPS Quick Log** | `~/logs/gps_quick.jsonl` |
| **GPS Correlations** | `~/logs/gps_correlations.db` (legacy) |
| **Exports** | `~/exports/subghz_wardriving/` |
| **Bruce Captures** | `~/projects/device-integration/bruce/data/captures/` |
| **Quick Capture Script** | `~/python/subghz_wardriving/quick_capture.py` |
| **Exporter Module** | `~/python/subghz_wardriving/exporter.py` |
| **Database Module** | `~/python/subghz_wardriving/database.py` |
## Integration with Existing GPS Infrastructure
This system integrates with your existing GPS wardriving infrastructure:
### Existing Systems
1. **`gps` command** - CLI tool for GPS operations
2. **`gps_service.py`** - Background GPS logging (158.4s interval)
3. **`smart_gps_detector.py`** - Auto-detection and file renaming
4. **`realtime_gps_attribution.py`** - Real-time GPS attribution
### New Additions
1. **`quick_capture.py`** - Quick GPS+RF pairing
2. **Wigle export format** - Added to `exporter.py`
3. **`gps wigle-export`** - New command in `gps` script
4. **Standardized database** - Uses existing `subghz_wardriving.db`
## Advanced Features
### GPS Interpolation
When RF captures occur between GPS readings, the system interpolates GPS coordinates based on:
- Time gap between captures
- Distance traveled
- Estimated speed
- Interpolation confidence score
### Session Management
```python
from subghz_wardriving.database import SubGHzWardrivingDB
db = SubGHzWardrivingDB()
# Create session
db.create_session("my_session", device_info="Bruce T-Embed", notes="Downtown survey")
# End session (calculates statistics)
db.end_session("my_session")
# Get summary
summary = db.get_session_summary("my_session")
print(f"Signals: {summary['session']['total_signals']}")
print(f"Distance: {summary['session']['total_distance_km']} km")
```
### Filtering Exports
```python
from subghz_wardriving.exporter import SubGHzExporter, ExportConfig
exporter = SubGHzExporter()
# Export only 433 MHz signals
config = ExportConfig(
format='wigle',
frequency_filter=(433_000_000, 434_000_000), # 433-434 MHz
confidence_threshold=0.8, # High confidence only
protocol_filter='RAW'
)
result = exporter.export_session(config)
```
## Comparison to WiFi Wigle
| Feature | WiFi Wigle | SubGHz Wigle |
|---------|------------|--------------|
| **Signal ID** | BSSID (MAC address) | SignalHash (freq+protocol) |
| **Signal Name** | SSID | Protocol name |
| **Frequency** | WiFi channels (2.4/5 GHz) | SubGHz (300-928 MHz) |
| **Strength** | RSSI (dBm) | Not available from .sub |
| **Device** | WiFi adapter | Bruce T-Embed CC1101 |
| **Format** | CSV | CSV (Wigle-compatible) |
| **Location** | GPS | GPS |
| **Aggregation** | By BSSID | By SignalHash |
## Future Enhancements
1. **RSSI Integration** - If Bruce firmware provides signal strength
2. **Real-time Export** - Live streaming to Wigle format
3. **Map Visualization** - Plot signals on map
4. **Protocol Decoding** - Decode known protocols (garage doors, car keys, etc.)
5. **Community Database** - Share SubGHz signal maps
6. **Cross-device Correlation** - Combine with Pineapple WiFi wardriving
## Troubleshooting
### No GPS Fix
```bash
# Check GPS status
gps status
# Try network provider
termux-location -p network
# Use test coordinates
# Edit gps script to add test coordinate option
```
### Database Locked
```bash
# Check for running processes
ps aux | grep python | grep subghz
# Close database connections
pkill -f subghz_wardriving
```
### Export Fails
```bash
# Check database exists
ls -lh ~/logs/subghz_wardriving.db
# Check export directory
mkdir -p ~/exports/subghz_wardriving
# Test export directly
python3 -m subghz_wardriving.exporter
```
## Resources
- **Bruce Firmware:** https://github.com/pr3y/Bruce
- **Flipper SubGHz Format:** https://docs.flipperzero.one/sub-ghz
- **Wigle.net:** https://wigle.net/
- **CC1101 Datasheet:** https://www.ti.com/product/CC1101
---
**Created:** 2026-01-13
**System Version:** 1.0
**Database:** ~/logs/subghz_wardriving.db
**Latest GPS:** 34.07362505, -118.35955673 (24.5m accuracy)
+178
View File
@@ -0,0 +1,178 @@
#!/data/data/com.termux/files/usr/bin/python3
"""
Quick GPS+RF Capture - Uses existing wardriving infrastructure
Captures current GPS and associates it with a Bruce .sub file
"""
import json
import sys
import subprocess
import hashlib
from datetime import datetime, timezone
from pathlib import Path
# Import existing infrastructure
from database import SubGHzWardrivingDB
def get_current_gps(timeout=15):
"""Get current GPS using termux-location."""
try:
cmd = ["termux-location", "-p", "gps"]
result = subprocess.run(
cmd,
capture_output=True,
text=True,
timeout=timeout
)
if result.returncode == 0 and result.stdout.strip():
gps_data = json.loads(result.stdout.strip())
if 'latitude' in gps_data and 'longitude' in gps_data:
print(f"✓ GPS Location: {gps_data['latitude']:.6f}, {gps_data['longitude']:.6f}")
print(f" Accuracy: {gps_data.get('accuracy', 'N/A')}m")
return gps_data
except subprocess.TimeoutExpired:
print(f"✗ GPS timeout after {timeout}s")
except Exception as e:
print(f"✗ GPS error: {e}")
return None
def parse_sub_file(sub_file):
"""Parse Bruce .sub file to extract RF metadata."""
metadata = {}
try:
with open(sub_file, 'r') as f:
for line in f:
line = line.strip()
if not line or not ':' in line:
continue
key, value = line.split(':', 1)
key = key.strip()
value = value.strip()
if key == 'Frequency':
metadata['frequency_hz'] = int(value)
elif key == 'Protocol':
metadata['protocol'] = value
elif key == 'Preset':
metadata['preset'] = value
return metadata if metadata else None
except Exception as e:
print(f"✗ Error parsing {sub_file}: {e}")
return None
def calculate_file_hash(file_path):
"""Calculate MD5 hash of file."""
md5 = hashlib.md5()
with open(file_path, 'rb') as f:
for chunk in iter(lambda: f.read(4096), b''):
md5.update(chunk)
return md5.hexdigest()
def main():
if len(sys.argv) < 2:
print("Usage: quick_capture.py <bruce_file.sub> [session_id]")
print("\nExample:")
print(" python3 quick_capture.py raw10.sub")
print(" python3 quick_capture.py ~/projects/device-integration/bruce/data/captures/raw10.sub mysession")
return 1
sub_file = Path(sys.argv[1])
session_id = sys.argv[2] if len(sys.argv) > 2 else f"quick_{datetime.now().strftime('%Y%m%d_%H%M%S')}"
if not sub_file.exists():
print(f"✗ File not found: {sub_file}")
return 1
print(f"\n📡 SubGHz GPS Capture")
print(f"━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━")
print(f"Bruce File: {sub_file.name}")
print(f"Session: {session_id}")
print()
# Get GPS
print("🛰️ Getting GPS location...")
gps_data = get_current_gps()
if not gps_data:
print("\n✗ Failed to get GPS - trying to use latest from log...")
gps_log = Path.home() / "logs" / "gps_quick.jsonl"
if gps_log.exists():
with open(gps_log, 'r') as f:
last_line = f.readlines()[-1]
gps_data = json.loads(last_line.strip())
print(f"✓ Using latest GPS: {gps_data['latitude']:.6f}, {gps_data['longitude']:.6f}")
else:
print("✗ No GPS data available")
return 1
# Parse RF file
print("\n📻 Parsing RF file...")
rf_metadata = parse_sub_file(sub_file)
if not rf_metadata:
print("✗ Failed to parse RF file")
return 1
print(f"✓ Frequency: {rf_metadata.get('frequency_hz', 'N/A')} Hz ({rf_metadata.get('frequency_hz', 0) / 1e6:.3f} MHz)")
print(f" Protocol: {rf_metadata.get('protocol', 'N/A')}")
print(f" Preset: {rf_metadata.get('preset', 'N/A')}")
# Initialize database
print("\n💾 Storing to database...")
db = SubGHzWardrivingDB()
# Create session if it doesn't exist
db.create_session(session_id, device_info="Bruce T-Embed CC1101", notes="Quick GPS capture")
# Add GPS track point
db.add_gps_point(
session_id=session_id,
latitude=gps_data['latitude'],
longitude=gps_data['longitude'],
altitude=gps_data.get('altitude'),
accuracy=gps_data.get('accuracy'),
speed=gps_data.get('speed'),
provider=gps_data.get('provider', 'gps')
)
# Add signal capture
timestamp = datetime.now(timezone.utc).isoformat()
capture_id = f"{session_id}_{sub_file.stem}_{int(datetime.now().timestamp())}"
db.add_signal(
session_id=session_id,
capture_id=capture_id,
timestamp=timestamp,
bruce_filename=sub_file.name,
bruce_file_path=str(sub_file),
file_size_bytes=sub_file.stat().st_size,
frequency_hz=rf_metadata.get('frequency_hz'),
protocol=rf_metadata.get('protocol'),
gps_latitude=gps_data['latitude'],
gps_longitude=gps_data['longitude'],
gps_accuracy_meters=gps_data.get('accuracy'),
interpolated=False,
interpolation_confidence=1.0,
file_hash_md5=calculate_file_hash(sub_file)
)
print(f"✓ Stored signal: {capture_id}")
print(f"\n✅ Capture paired successfully!")
print(f" Database: ~/logs/subghz_wardriving.db")
print(f" Export with: python3 -m subghz_wardriving.exporter")
print()
return 0
if __name__ == '__main__':
sys.exit(main())
@@ -0,0 +1,3 @@
WigleWifi-1.4,appRelease=SubGHz-Wardrive-1.0,model=BruceT-Embed,release=Android,device=Phone,display=SubGHzWardrive,board=termux,brand=custom
SignalHash,Frequency(MHz),Protocol,RSSI(dBm),Latitude,Longitude,Altitude,Accuracy,FirstSeen,LastSeen,CaptureCount,Interpolated,Confidence,BruceFile,SessionID
8a3f2d1c4e5b6a7f,433.920,RAW,,34.073625,-118.359557,,24.5,2026-01-13T19:17:51Z,2026-01-13T19:17:51Z,1,false,1.0,raw10.sub,test_session
1 WigleWifi-1.4,appRelease=SubGHz-Wardrive-1.0,model=BruceT-Embed,release=Android,device=Phone,display=SubGHzWardrive,board=termux,brand=custom
2 SignalHash,Frequency(MHz),Protocol,RSSI(dBm),Latitude,Longitude,Altitude,Accuracy,FirstSeen,LastSeen,CaptureCount,Interpolated,Confidence,BruceFile,SessionID
3 8a3f2d1c4e5b6a7f,433.920,RAW,,34.073625,-118.359557,,24.5,2026-01-13T19:17:51Z,2026-01-13T19:17:51Z,1,false,1.0,raw10.sub,test_session
+126 -10
View File
@@ -4,13 +4,22 @@ GigLez FastAPI Application - Simplified Version
Runs without database requirement for testing web interface Runs without database requirement for testing web interface
""" """
from fastapi import FastAPI, Request import json
import sys
from pathlib import Path
from typing import List
from fastapi import FastAPI, Request, UploadFile, File, Form
from fastapi.middleware.cors import CORSMiddleware from fastapi.middleware.cors import CORSMiddleware
from fastapi.middleware.gzip import GZipMiddleware from fastapi.middleware.gzip import GZipMiddleware
from fastapi.responses import HTMLResponse from fastapi.responses import HTMLResponse
from fastapi.staticfiles import StaticFiles from fastapi.staticfiles import StaticFiles
from fastapi.templating import Jinja2Templates from fastapi.templating import Jinja2Templates
from pathlib import Path
# Add project root to path for imports
sys.path.insert(0, str(Path(__file__).parent.parent.parent))
from src.parser.sub_parser import SubFileParser
from src.parser.gps_extractor import GPSFilenameExtractor
# ============================================================================= # =============================================================================
# APPLICATION INSTANCE # APPLICATION INSTANCE
@@ -29,6 +38,10 @@ BASE_DIR = Path(__file__).parent.parent.parent
app.mount("/static", StaticFiles(directory=str(BASE_DIR / "static")), name="static") app.mount("/static", StaticFiles(directory=str(BASE_DIR / "static")), name="static")
templates = Jinja2Templates(directory=str(BASE_DIR / "templates")) templates = Jinja2Templates(directory=str(BASE_DIR / "templates"))
# In-memory storage for uploaded captures (simplified mode)
captures_storage = []
upload_counter = 0
# ============================================================================= # =============================================================================
# MIDDLEWARE # MIDDLEWARE
@@ -85,10 +98,10 @@ async def api_root():
@app.get("/api/v1/query/captures") @app.get("/api/v1/query/captures")
async def get_captures(): async def get_captures():
"""Mock endpoint - return empty captures for now""" """Return stored captures from in-memory storage"""
return { return {
"captures": [], "captures": captures_storage,
"total": 0, "total": len(captures_storage),
"page": 1, "page": 1,
"page_size": 100 "page_size": 100
} }
@@ -96,17 +109,120 @@ async def get_captures():
@app.get("/api/v1/stats/summary") @app.get("/api/v1/stats/summary")
async def get_stats(): async def get_stats():
"""Mock endpoint - return zero stats for now""" """Return stats from in-memory storage"""
# Calculate frequency distribution
freq_dist = {}
for capture in captures_storage:
freq = capture.get("frequency", 0)
if freq > 0:
freq_dist[freq] = freq_dist.get(freq, 0) + 1
# Count unique protocols as proxy for unique devices
unique_protocols = len(set(c.get("protocol", "Unknown") for c in captures_storage))
return { return {
"total_captures": 0, "total_captures": len(captures_storage),
"unique_devices": 0, "unique_devices": unique_protocols,
"coverage_area_km2": 0, "coverage_area_km2": 0,
"total_contributors": 0, "total_contributors": 1 if len(captures_storage) > 0 else 0,
"frequency_distribution": {}, "frequency_distribution": freq_dist,
"captures_timeline": [] "captures_timeline": []
} }
@app.post("/api/v1/captures/upload")
async def upload_captures(
files: List[UploadFile] = File(...),
manifest: str = Form(...)
):
"""
Mock upload endpoint - parses files and returns success
Does not save to database (simplified version)
"""
import tempfile
import os
try:
# Parse manifest
manifest_data = json.loads(manifest)
# Initialize parsers
sub_parser = SubFileParser()
gps_extractor = GPSFilenameExtractor()
successful = []
failed = []
# Process each file
for file in files:
temp_path = None
try:
# Read file content
content = await file.read()
# Save to temporary file for parsing
with tempfile.NamedTemporaryFile(mode='wb', delete=False, suffix='.sub') as temp_file:
temp_path = temp_file.name
temp_file.write(content)
# Parse .sub file
metadata = sub_parser.parse(temp_path)
# Try to extract GPS from filename
gps_coords = gps_extractor.extract(file.filename)
# Use GPS from manifest or filename
global upload_counter
upload_counter += 1
capture_info = {
"id": upload_counter,
"filename": file.filename,
"frequency": metadata.frequency if hasattr(metadata, 'frequency') else 0,
"protocol": metadata.protocol if hasattr(metadata, 'protocol') else "RAW",
"preset": metadata.preset if hasattr(metadata, 'preset') else "Unknown",
"latitude": gps_coords.latitude if gps_coords else manifest_data.get("captures", [{}])[0].get("latitude"),
"longitude": gps_coords.longitude if gps_coords else manifest_data.get("captures", [{}])[0].get("longitude"),
"gps_source": gps_coords.source if gps_coords else "manual",
"timestamp": manifest_data.get("captures", [{}])[0].get("timestamp", ""),
}
# Store in memory
captures_storage.append(capture_info)
successful.append(capture_info)
except Exception as e:
failed.append({
"filename": file.filename,
"error": str(e)
})
finally:
# Clean up temp file
if temp_path and os.path.exists(temp_path):
os.unlink(temp_path)
return {
"success": True,
"message": f"Processed {len(successful)} files successfully",
"successful": successful,
"failed": failed,
"total_uploaded": len(files),
"total_successful": len(successful),
"total_failed": len(failed)
}
except Exception as e:
return {
"success": False,
"message": f"Upload failed: {str(e)}",
"successful": [],
"failed": [],
"total_uploaded": 0,
"total_successful": 0,
"total_failed": 0
}
# ============================================================================= # =============================================================================
# RUN WITH UVICORN (for development) # RUN WITH UVICORN (for development)
# ============================================================================= # =============================================================================
+241
View File
@@ -0,0 +1,241 @@
"""
GPS Coordinate Extraction from Filenames
Automatically extracts GPS coordinates from filename patterns
"""
import re
from pathlib import Path
from typing import Optional, Dict, Tuple
from dataclasses import dataclass
@dataclass
class GPSCoordinates:
"""GPS coordinates extracted from filename"""
latitude: float
longitude: float
source: str = "filename"
accuracy: Optional[float] = None
altitude: Optional[float] = None
def to_dict(self) -> Dict:
"""Convert to dictionary"""
return {
'latitude': self.latitude,
'longitude': self.longitude,
'source': self.source,
'accuracy': self.accuracy,
'altitude': self.altitude
}
class GPSFilenameExtractor:
"""
Extract GPS coordinates from various filename patterns
Supported patterns:
1. Decimal degrees: 34.0478N_118.2348W_timestamp_name.sub
2. Signed decimal: -34.0478_118.2348_timestamp_name.sub
3. DMS format: 34d02m52sN_118d14m05sW_name.sub
4. Compact: lat34.0478lon-118.2348_name.sub
"""
# Pattern 1: Decimal degrees with N/S/E/W (most common)
PATTERN_NSEW = re.compile(
r'(\d+\.?\d*)([NS])_(\d+\.?\d*)([EW])',
re.IGNORECASE
)
# Pattern 2: Signed decimal degrees
PATTERN_SIGNED = re.compile(
r'(-?\d+\.?\d+)_(-?\d+\.?\d+)'
)
# Pattern 3: lat/lon prefix format
PATTERN_LATLON = re.compile(
r'lat(-?\d+\.?\d+)lon(-?\d+\.?\d+)',
re.IGNORECASE
)
# Pattern 4: DMS (Degrees Minutes Seconds)
PATTERN_DMS = re.compile(
r'(\d+)d(\d+)m(\d+\.?\d*)s([NS])_(\d+)d(\d+)m(\d+\.?\d*)s([EW])',
re.IGNORECASE
)
def extract(self, filename: str) -> Optional[GPSCoordinates]:
"""
Extract GPS coordinates from filename
Args:
filename: Filename to parse (with or without path)
Returns:
GPSCoordinates if found, None otherwise
"""
# Get just the filename without path
basename = Path(filename).name
# Try each pattern
coords = (
self._try_nsew(basename) or
self._try_latlon(basename) or
self._try_signed(basename) or
self._try_dms(basename)
)
return coords
def _try_nsew(self, filename: str) -> Optional[GPSCoordinates]:
"""Try N/S/E/W pattern: 34.0478N_118.2348W"""
match = self.PATTERN_NSEW.search(filename)
if not match:
return None
lat_val, lat_dir, lon_val, lon_dir = match.groups()
# Convert to float
lat = float(lat_val)
lon = float(lon_val)
# Apply direction (N=positive, S=negative, E=positive, W=negative)
if lat_dir.upper() == 'S':
lat = -lat
if lon_dir.upper() == 'W':
lon = -lon
# Validate ranges
if not self._validate_coordinates(lat, lon):
return None
return GPSCoordinates(
latitude=lat,
longitude=lon,
source="filename_nsew"
)
def _try_signed(self, filename: str) -> Optional[GPSCoordinates]:
"""Try signed decimal: -34.0478_118.2348"""
match = self.PATTERN_SIGNED.search(filename)
if not match:
return None
lat, lon = match.groups()
lat = float(lat)
lon = float(lon)
if not self._validate_coordinates(lat, lon):
return None
return GPSCoordinates(
latitude=lat,
longitude=lon,
source="filename_signed"
)
def _try_latlon(self, filename: str) -> Optional[GPSCoordinates]:
"""Try lat/lon prefix: lat34.0478lon-118.2348"""
match = self.PATTERN_LATLON.search(filename)
if not match:
return None
lat, lon = match.groups()
lat = float(lat)
lon = float(lon)
if not self._validate_coordinates(lat, lon):
return None
return GPSCoordinates(
latitude=lat,
longitude=lon,
source="filename_latlon"
)
def _try_dms(self, filename: str) -> Optional[GPSCoordinates]:
"""Try DMS format: 34d02m52sN_118d14m05sW"""
match = self.PATTERN_DMS.search(filename)
if not match:
return None
lat_deg, lat_min, lat_sec, lat_dir, lon_deg, lon_min, lon_sec, lon_dir = match.groups()
# Convert DMS to decimal degrees
lat = self._dms_to_decimal(
int(lat_deg), int(lat_min), float(lat_sec)
)
lon = self._dms_to_decimal(
int(lon_deg), int(lon_min), float(lon_sec)
)
# Apply direction
if lat_dir.upper() == 'S':
lat = -lat
if lon_dir.upper() == 'W':
lon = -lon
if not self._validate_coordinates(lat, lon):
return None
return GPSCoordinates(
latitude=lat,
longitude=lon,
source="filename_dms"
)
@staticmethod
def _dms_to_decimal(degrees: int, minutes: int, seconds: float) -> float:
"""Convert DMS to decimal degrees"""
return degrees + (minutes / 60.0) + (seconds / 3600.0)
@staticmethod
def _validate_coordinates(lat: float, lon: float) -> bool:
"""Validate coordinate ranges"""
return -90 <= lat <= 90 and -180 <= lon <= 180
def extract_gps_from_filename(filename: str) -> Optional[Dict]:
"""
Convenience function to extract GPS from filename
Args:
filename: Filename to parse
Returns:
Dictionary with GPS data or None
Example:
>>> extract_gps_from_filename("34.0478N_118.2348W_capture.sub")
{'latitude': 34.0478, 'longitude': -118.2348, 'source': 'filename_nsew'}
"""
extractor = GPSFilenameExtractor()
coords = extractor.extract(filename)
return coords.to_dict() if coords else None
# Example usage and testing
if __name__ == "__main__":
test_filenames = [
"34.0478N_118.2348W_1637_raw_8.sub", # N/S/E/W format
"34.0478N_118.2349W_1351_raw_10.sub",
"lat34.0478lon-118.2348_test.sub", # lat/lon prefix
"-34.0478_118.2348_capture.sub", # Signed decimal
"34d02m52sN_118d14m05sW_test.sub", # DMS format
"raw_7.sub", # No GPS
]
extractor = GPSFilenameExtractor()
print("GPS Filename Extraction Test")
print("=" * 60)
for filename in test_filenames:
coords = extractor.extract(filename)
if coords:
print(f"{filename}")
print(f" Lat: {coords.latitude:.6f}, Lon: {coords.longitude:.6f}")
print(f" Source: {coords.source}")
else:
print(f"⏭️ {filename} - No GPS found")
print()
+177 -12
View File
@@ -1,6 +1,7 @@
// GigLez - Upload Functionality // GigLez - Upload Functionality
let selectedFiles = []; let selectedFiles = [];
let autoDetectedGPS = null; // Store GPS detected from filenames
// Initialize upload functionality // Initialize upload functionality
document.addEventListener('DOMContentLoaded', () => { document.addEventListener('DOMContentLoaded', () => {
@@ -8,6 +9,90 @@ document.addEventListener('DOMContentLoaded', () => {
setupFileInput(); setupFileInput();
}); });
// GPS Extraction from Filenames
// Matches patterns like: 34.0478N_118.2349W_1650_test_raw.sub
class GPSFilenameExtractor {
constructor() {
// Pattern 1: Decimal degrees with N/S/E/W (most common)
this.PATTERN_NSEW = /(\d+\.?\d*)([NS])_(\d+\.?\d*)([EW])/i;
// Pattern 2: lat/lon prefix format
this.PATTERN_LATLON = /lat(-?\d+\.?\d+)lon(-?\d+\.?\d+)/i;
// Pattern 3: Signed decimal degrees
this.PATTERN_SIGNED = /(-?\d+\.?\d+)_(-?\d+\.?\d+)/;
}
extract(filename) {
// Try each pattern
let coords = this.tryNSEW(filename) ||
this.tryLatLon(filename) ||
this.trySigned(filename);
return coords;
}
tryNSEW(filename) {
const match = filename.match(this.PATTERN_NSEW);
if (!match) return null;
let lat = parseFloat(match[1]);
let lon = parseFloat(match[3]);
const latDir = match[2].toUpperCase();
const lonDir = match[4].toUpperCase();
// Apply direction (N=positive, S=negative, E=positive, W=negative)
if (latDir === 'S') lat = -lat;
if (lonDir === 'W') lon = -lon;
if (!this.validateCoordinates(lat, lon)) return null;
return {
latitude: lat,
longitude: lon,
source: 'filename_nsew'
};
}
tryLatLon(filename) {
const match = filename.match(this.PATTERN_LATLON);
if (!match) return null;
const lat = parseFloat(match[1]);
const lon = parseFloat(match[2]);
if (!this.validateCoordinates(lat, lon)) return null;
return {
latitude: lat,
longitude: lon,
source: 'filename_latlon'
};
}
trySigned(filename) {
const match = filename.match(this.PATTERN_SIGNED);
if (!match) return null;
const lat = parseFloat(match[1]);
const lon = parseFloat(match[2]);
if (!this.validateCoordinates(lat, lon)) return null;
return {
latitude: lat,
longitude: lon,
source: 'filename_signed'
};
}
validateCoordinates(lat, lon) {
return lat >= -90 && lat <= 90 && lon >= -180 && lon <= 180;
}
}
const gpsExtractor = new GPSFilenameExtractor();
function setupDropZone() { function setupDropZone() {
const dropZone = document.getElementById('drop-zone'); const dropZone = document.getElementById('drop-zone');
@@ -51,6 +136,23 @@ function addFiles(files) {
// Add new files // Add new files
selectedFiles.push(...files); selectedFiles.push(...files);
// Try to extract GPS from first file with coordinates
if (!autoDetectedGPS) {
for (const file of files) {
const coords = gpsExtractor.extract(file.name);
if (coords) {
autoDetectedGPS = coords;
// Auto-populate GPS fields
document.getElementById('default-lat').value = coords.latitude.toFixed(6);
document.getElementById('default-lon').value = coords.longitude.toFixed(6);
// Show notification
showGPSDetectionNotification(file.name, coords);
break;
}
}
}
// Show file list // Show file list
document.getElementById('file-list').style.display = 'block'; document.getElementById('file-list').style.display = 'block';
@@ -58,20 +160,72 @@ function addFiles(files) {
renderFileList(); renderFileList();
} }
function showGPSDetectionNotification(filename, coords) {
const notification = document.createElement('div');
notification.style.cssText = `
position: fixed;
top: 20px;
right: 20px;
background: #10b981;
color: white;
padding: 1rem 1.5rem;
border-radius: 0.5rem;
box-shadow: 0 4px 6px rgba(0, 0, 0, 0.1);
z-index: 10000;
max-width: 400px;
animation: slideIn 0.3s ease-out;
`;
notification.innerHTML = `
<strong>📍 GPS Auto-Detected!</strong><br>
<small>From: ${filename}</small><br>
<small>Coordinates: ${coords.latitude.toFixed(6)}, ${coords.longitude.toFixed(6)}</small>
`;
document.body.appendChild(notification);
// Add animation
const style = document.createElement('style');
style.textContent = `
@keyframes slideIn {
from { transform: translateX(400px); opacity: 0; }
to { transform: translateX(0); opacity: 1; }
}
`;
document.head.appendChild(style);
// Remove after 5 seconds
setTimeout(() => {
notification.style.animation = 'slideIn 0.3s ease-out reverse';
setTimeout(() => notification.remove(), 300);
}, 5000);
}
function renderFileList() { function renderFileList() {
const container = document.getElementById('files-container'); const container = document.getElementById('files-container');
container.innerHTML = selectedFiles.map((file, index) => ` container.innerHTML = selectedFiles.map((file, index) => {
<div class="file-item" data-index="${index}"> const coords = gpsExtractor.extract(file.name);
<div class="file-info"> const hasGPS = coords !== null;
<div class="file-name">${file.name}</div>
<div class="file-meta">${formatFileSize(file.size)}</div> return `
<div class="file-item" data-index="${index}">
<div class="file-info">
<div class="file-name">
${file.name}
${hasGPS ? '<span style="color: #10b981; margin-left: 0.5rem;">📍 GPS</span>' : ''}
</div>
<div class="file-meta">
${formatFileSize(file.size)}
${hasGPS ? ` | ${coords.latitude.toFixed(4)}, ${coords.longitude.toFixed(4)}` : ''}
</div>
</div>
<div class="file-actions">
<button type="button" class="btn btn-secondary" onclick="removeFile(${index})">Remove</button>
</div>
</div> </div>
<div class="file-actions"> `;
<button type="button" class="btn btn-secondary" onclick="removeFile(${index})">Remove</button> }).join('');
</div>
</div>
`).join('');
} }
function removeFile(index) { function removeFile(index) {
@@ -86,6 +240,7 @@ function removeFile(index) {
function clearFiles() { function clearFiles() {
selectedFiles = []; selectedFiles = [];
autoDetectedGPS = null; // Reset auto-detected GPS
document.getElementById('file-list').style.display = 'none'; document.getElementById('file-list').style.display = 'none';
document.getElementById('file-input').value = ''; document.getElementById('file-input').value = '';
} }
@@ -96,12 +251,19 @@ async function uploadFiles() {
return; return;
} }
// Validate GPS coordinates // Validate GPS coordinates (allow auto-detected or manual entry)
const lat = parseFloat(document.getElementById('default-lat').value); const lat = parseFloat(document.getElementById('default-lat').value);
const lon = parseFloat(document.getElementById('default-lon').value); const lon = parseFloat(document.getElementById('default-lon').value);
if (isNaN(lat) || isNaN(lon)) { if (isNaN(lat) || isNaN(lon)) {
alert('Please provide valid GPS coordinates'); // Check if any file has GPS in filename
const hasGPSInFilename = selectedFiles.some(file => gpsExtractor.extract(file.name) !== null);
if (hasGPSInFilename) {
alert('GPS detected in filename but not populated. Please refresh and try again.');
} else {
alert('Please provide GPS coordinates (manually or via filename like: 34.0478N_118.2349W_filename.sub)');
}
return; return;
} }
@@ -236,6 +398,9 @@ function resetUpload() {
document.getElementById('default-lat').value = ''; document.getElementById('default-lat').value = '';
document.getElementById('default-lon').value = ''; document.getElementById('default-lon').value = '';
document.getElementById('session-uuid').value = ''; document.getElementById('session-uuid').value = '';
// Reset auto-detected GPS
autoDetectedGPS = null;
} }
function useCurrentLocation() { function useCurrentLocation() {
+176
View File
@@ -0,0 +1,176 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Test GPS Extraction</title>
<style>
body {
font-family: system-ui, -apple-system, sans-serif;
max-width: 800px;
margin: 50px auto;
padding: 20px;
background: #f5f5f5;
}
.test-case {
background: white;
padding: 15px;
margin: 10px 0;
border-radius: 8px;
border-left: 4px solid #2563eb;
}
.filename {
font-family: monospace;
background: #f0f0f0;
padding: 5px 10px;
border-radius: 4px;
margin: 5px 0;
}
.result {
margin-top: 10px;
padding: 10px;
background: #f0fdf4;
border-left: 3px solid #10b981;
border-radius: 4px;
}
.result.fail {
background: #fef2f2;
border-left-color: #ef4444;
}
</style>
</head>
<body>
<h1>GPS Filename Extraction Test</h1>
<p>Testing the same patterns as Python implementation</p>
<div id="results"></div>
<script>
// Copy the GPS extraction code from upload.js
class GPSFilenameExtractor {
constructor() {
this.PATTERN_NSEW = /(\d+\.?\d*)([NS])_(\d+\.?\d*)([EW])/i;
this.PATTERN_LATLON = /lat(-?\d+\.?\d+)lon(-?\d+\.?\d+)/i;
this.PATTERN_SIGNED = /(-?\d+\.?\d+)_(-?\d+\.?\d+)/;
}
extract(filename) {
let coords = this.tryNSEW(filename) ||
this.tryLatLon(filename) ||
this.trySigned(filename);
return coords;
}
tryNSEW(filename) {
const match = filename.match(this.PATTERN_NSEW);
if (!match) return null;
let lat = parseFloat(match[1]);
let lon = parseFloat(match[3]);
const latDir = match[2].toUpperCase();
const lonDir = match[4].toUpperCase();
if (latDir === 'S') lat = -lat;
if (lonDir === 'W') lon = -lon;
if (!this.validateCoordinates(lat, lon)) return null;
return {
latitude: lat,
longitude: lon,
source: 'filename_nsew'
};
}
tryLatLon(filename) {
const match = filename.match(this.PATTERN_LATLON);
if (!match) return null;
const lat = parseFloat(match[1]);
const lon = parseFloat(match[2]);
if (!this.validateCoordinates(lat, lon)) return null;
return {
latitude: lat,
longitude: lon,
source: 'filename_latlon'
};
}
trySigned(filename) {
const match = filename.match(this.PATTERN_SIGNED);
if (!match) return null;
const lat = parseFloat(match[1]);
const lon = parseFloat(match[2]);
if (!this.validateCoordinates(lat, lon)) return null;
return {
latitude: lat,
longitude: lon,
source: 'filename_signed'
};
}
validateCoordinates(lat, lon) {
return lat >= -90 && lat <= 90 && lon >= -180 && lon <= 180;
}
}
// Test cases
const testCases = [
{ filename: "34.0478N_118.2348W_1637_raw_8.sub", expected: { lat: 34.0478, lon: -118.2348 } },
{ filename: "34.0478N_118.2349W_1351_raw_10.sub", expected: { lat: 34.0478, lon: -118.2349 } },
{ filename: "34.0478N_118.2349W_1650_test_raw.sub", expected: { lat: 34.0478, lon: -118.2349 } },
{ filename: "lat34.0478lon-118.2348_test.sub", expected: { lat: 34.0478, lon: -118.2348 } },
{ filename: "-34.0478_118.2348_capture.sub", expected: { lat: -34.0478, lon: 118.2348 } },
{ filename: "raw_7.sub", expected: null },
{ filename: "raw_8.sub", expected: null }
];
const extractor = new GPSFilenameExtractor();
const resultsDiv = document.getElementById('results');
testCases.forEach((test, index) => {
const coords = extractor.extract(test.filename);
const testDiv = document.createElement('div');
testDiv.className = 'test-case';
let success = false;
let resultHTML = '';
if (test.expected === null) {
// Expect no GPS
success = coords === null;
resultHTML = coords === null ?
`<div class="result">✅ Correctly detected: No GPS in filename</div>` :
`<div class="result fail">❌ False positive: ${coords.latitude}, ${coords.longitude}</div>`;
} else {
// Expect GPS
if (coords) {
const latMatch = Math.abs(coords.latitude - test.expected.lat) < 0.0001;
const lonMatch = Math.abs(coords.longitude - test.expected.lon) < 0.0001;
success = latMatch && lonMatch;
resultHTML = success ?
`<div class="result">✅ GPS Detected: ${coords.latitude.toFixed(6)}, ${coords.longitude.toFixed(6)} (${coords.source})</div>` :
`<div class="result fail">❌ Wrong coordinates: Got ${coords.latitude}, ${coords.longitude} | Expected ${test.expected.lat}, ${test.expected.lon}</div>`;
} else {
resultHTML = `<div class="result fail">❌ Failed to extract GPS (expected ${test.expected.lat}, ${test.expected.lon})</div>`;
}
}
testDiv.innerHTML = `
<div><strong>Test ${index + 1}</strong></div>
<div class="filename">${test.filename}</div>
${resultHTML}
`;
resultsDiv.appendChild(testDiv);
});
</script>
</body>
</html>