390 lines
12 KiB
Markdown
390 lines
12 KiB
Markdown
# GigLez Architecture Decisions (Based on Wigle Analysis)
|
|
|
|
## Overview
|
|
|
|
This document captures key architectural decisions for GigLez, informed by analysis of the Wigle Android wardriving client.
|
|
|
|
**Date**: 2026-01-12
|
|
**Status**: Proposed
|
|
**Source**: Wigle Analysis (/docs/wigle_analysis.md)
|
|
|
|
---
|
|
|
|
## Decision 1: Database Schema - PostgreSQL with PostGIS
|
|
|
|
**Context**: Need to store millions of RF captures with GPS coordinates for geospatial queries.
|
|
|
|
**Decision**: Use PostgreSQL + PostGIS with a 3-table structure:
|
|
- `captures` - Primary RF file data (like Wigle's `network` table)
|
|
- `capture_matches` - Device identification results (like Wigle's `location` table)
|
|
- `devices` - Known device signature database (reference data)
|
|
|
|
**Rationale**:
|
|
- Wigle's 3-table design (network/location/route) has scaled to **349M WiFi networks** and **billions of observations**
|
|
- PostGIS provides efficient geospatial indexing for radius/bounding box queries
|
|
- Separate matches table allows many-to-many relationships (one capture → multiple device matches)
|
|
|
|
**Alternatives Considered**:
|
|
- MongoDB: No ACID guarantees, weaker geospatial query performance
|
|
- Single table: Would duplicate device info, slower queries
|
|
|
|
**Implementation**:
|
|
```sql
|
|
CREATE TABLE captures (
|
|
file_hash text primary key,
|
|
latitude double precision not null,
|
|
longitude double precision not null,
|
|
geom geometry(Point, 4326), -- PostGIS spatial index
|
|
timestamp bigint not null,
|
|
frequency integer not null,
|
|
protocol text,
|
|
session_id text
|
|
);
|
|
CREATE INDEX idx_captures_geom ON captures USING GIST(geom);
|
|
```
|
|
|
|
---
|
|
|
|
## Decision 2: File Deduplication - SHA256 Hash as Primary Key
|
|
|
|
**Context**: Prevent duplicate .sub file submissions (same file uploaded multiple times).
|
|
|
|
**Decision**: Use SHA256 hash of file contents as primary key in `captures` table.
|
|
|
|
**Rationale**:
|
|
- Wigle uses BSSID (MAC address) as primary key for natural deduplication
|
|
- SHA256 provides **cryptographically strong** uniqueness guarantee
|
|
- Automatic deduplication at database level (INSERT fails on duplicate)
|
|
- No need for separate deduplication logic
|
|
|
|
**Alternatives Considered**:
|
|
- Auto-increment ID + separate hash check: More complex, requires application-level logic
|
|
- GPS + timestamp composite key: Could have legitimate duplicates
|
|
|
|
**Implementation**:
|
|
```python
|
|
import hashlib
|
|
|
|
def compute_file_hash(file_path: str) -> str:
|
|
sha256 = hashlib.sha256()
|
|
with open(file_path, 'rb') as f:
|
|
for chunk in iter(lambda: f.read(8192), b''):
|
|
sha256.update(chunk)
|
|
return sha256.hexdigest()
|
|
```
|
|
|
|
---
|
|
|
|
## Decision 3: Upload Format - JSON Manifest + Binary Files
|
|
|
|
**Context**: Need to upload .sub files with GPS metadata to server.
|
|
|
|
**Decision**: Use multipart/form-data with:
|
|
- JSON manifest (GPS coordinates, metadata)
|
|
- Binary .sub files (unmodified)
|
|
|
|
**Rationale**:
|
|
- Wigle uses CSV for structured data export - works for text but not binary files
|
|
- JSON provides structured metadata (per-file GPS, session info)
|
|
- Binary files preserve exact capture data (no encoding issues)
|
|
- Supports batch uploads naturally (multiple files in one request)
|
|
|
|
**Alternatives Considered**:
|
|
- CSV with base64-encoded files: Bloated (33% overhead), parsing complexity
|
|
- Separate API calls per file: Network inefficiency, no atomic batch
|
|
|
|
**Implementation**:
|
|
```json
|
|
{
|
|
"version": "GigLez-1.0",
|
|
"session_id": "uuid",
|
|
"captures": [
|
|
{
|
|
"file": "capture_001.sub",
|
|
"sha256": "abc123...",
|
|
"gps": {
|
|
"latitude": 40.7128,
|
|
"longitude": -74.0060,
|
|
"accuracy": 5.0,
|
|
"timestamp": "2025-01-11T20:30:00Z"
|
|
}
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## Decision 4: Background Processing - Async Task Queue
|
|
|
|
**Context**: .sub file parsing and device matching are CPU-intensive operations.
|
|
|
|
**Decision**: Use async background task queue (FastAPI BackgroundTasks or Celery/RQ) for:
|
|
- File parsing
|
|
- Device signature matching
|
|
- Database writes
|
|
|
|
**Rationale**:
|
|
- Wigle uses **background thread** for all database writes (THREAD_PRIORITY_BACKGROUND)
|
|
- Prevents API timeouts on large uploads
|
|
- Enables **immediate response** to user (202 Accepted)
|
|
- Allows **horizontal scaling** of workers
|
|
|
|
**Alternatives Considered**:
|
|
- Synchronous processing: Slow, API timeouts on large files
|
|
- Lambda functions: Cold start overhead, cost at scale
|
|
|
|
**Implementation**:
|
|
```python
|
|
from fastapi import BackgroundTasks
|
|
|
|
@app.post("/api/submit")
|
|
async def submit_capture(file: UploadFile, background_tasks: BackgroundTasks):
|
|
file_path = await save_upload(file)
|
|
task_id = uuid4()
|
|
|
|
background_tasks.add_task(
|
|
process_capture_pipeline,
|
|
file_path=file_path,
|
|
task_id=task_id
|
|
)
|
|
|
|
return {"status": "accepted", "task_id": str(task_id)}
|
|
```
|
|
|
|
---
|
|
|
|
## Decision 5: GPS Validation - Multi-Level Checks
|
|
|
|
**Context**: GPS coordinates can be invalid (null island, extreme accuracy, out of bounds).
|
|
|
|
**Decision**: Implement Wigle's multi-level GPS validation:
|
|
1. Bounds check (-90≤lat≤90, -180≤lon≤180)
|
|
2. Accuracy threshold (reject if >100m)
|
|
3. Null island check (reject lat=0, lon=0)
|
|
4. Timestamp validation (reject if 0)
|
|
|
|
**Rationale**:
|
|
- Wigle's horribleGps() function filters out **GPS noise** from hardware bugs
|
|
- Prevents database pollution with junk coordinates
|
|
- 100m threshold balances accuracy vs. rejection rate
|
|
|
|
**Alternatives Considered**:
|
|
- Accept all GPS: Database pollution, poor map quality
|
|
- Strict threshold (<10m): Too restrictive, rejects valid mobile captures
|
|
|
|
**Implementation**:
|
|
```python
|
|
def validate_gps(lat, lon, accuracy=None, timestamp=None):
|
|
if not (-90 <= lat <= 90) or not (-180 <= lon <= 180):
|
|
raise ValueError("GPS out of bounds")
|
|
if accuracy and accuracy > 100:
|
|
raise ValueError(f"GPS accuracy too low: {accuracy}m")
|
|
if lat == 0.0 and lon == 0.0:
|
|
raise ValueError("GPS at null island")
|
|
if timestamp == 0:
|
|
raise ValueError("Invalid timestamp")
|
|
```
|
|
|
|
---
|
|
|
|
## Decision 6: Upload Marker System - Incremental Sync
|
|
|
|
**Context**: Users may upload same database multiple times (after adding new captures).
|
|
|
|
**Decision**: Track last uploaded capture ID per user (like Wigle's PREF_DB_MARKER).
|
|
|
|
**Rationale**:
|
|
- Wigle's marker system enables **incremental sync** (only new data)
|
|
- Prevents re-uploading entire database on each sync
|
|
- Enables **resume after failure** (upload only remaining captures)
|
|
|
|
**Alternatives Considered**:
|
|
- Always upload all captures: Wasteful bandwidth, slow
|
|
- Client-side tracking only: Lost on app reinstall
|
|
|
|
**Implementation**:
|
|
```python
|
|
class UploadMarker(Base):
|
|
__tablename__ = 'upload_markers'
|
|
user_id = Column(String, primary_key=True)
|
|
last_uploaded_id = Column(Integer, default=0)
|
|
last_upload_time = Column(BigInteger)
|
|
|
|
def get_unuploaded_captures(user_id):
|
|
marker = db.query(UploadMarker).filter_by(user_id=user_id).first()
|
|
last_id = marker.last_uploaded_id if marker else 0
|
|
|
|
return db.query(Capture).filter(
|
|
Capture.id > last_id,
|
|
Capture.uploaded == False
|
|
).all()
|
|
```
|
|
|
|
---
|
|
|
|
## Decision 7: Session Management - Run ID System
|
|
|
|
**Context**: Group captures by wardriving session for batch operations and statistics.
|
|
|
|
**Decision**: Add `session_id` to captures table (like Wigle's `run_id`).
|
|
|
|
**Rationale**:
|
|
- Wigle's run_id groups **route points** by wardriving session
|
|
- Enables batch operations (upload session, view session stats)
|
|
- Useful for statistics (captures per session, coverage area)
|
|
- Supports partial uploads (session-by-session)
|
|
|
|
**Alternatives Considered**:
|
|
- No grouping: Hard to track wardriving runs, no batch operations
|
|
- Timestamp-based grouping: Unreliable (gaps in time)
|
|
|
|
**Implementation**:
|
|
```sql
|
|
CREATE TABLE sessions (
|
|
session_id text primary key,
|
|
user_id text not null,
|
|
start_time bigint not null,
|
|
end_time bigint,
|
|
capture_count integer default 0,
|
|
uploaded boolean default false
|
|
);
|
|
|
|
ALTER TABLE captures ADD COLUMN session_id text REFERENCES sessions(session_id);
|
|
CREATE INDEX idx_captures_session ON captures(session_id);
|
|
```
|
|
|
|
---
|
|
|
|
## Decision 8: Performance - Transaction Batching
|
|
|
|
**Context**: Writing captures one-at-a-time is slow (disk I/O overhead).
|
|
|
|
**Decision**: Batch database writes in transactions (up to 512 operations).
|
|
|
|
**Rationale**:
|
|
- Wigle batches up to **512 DB updates** per transaction
|
|
- Reduces disk I/O by 100x+ (one fsync per batch vs. per operation)
|
|
- Critical for write-heavy workloads (wardriving generates 1000s of observations)
|
|
|
|
**Alternatives Considered**:
|
|
- Individual transactions: 100x slower, disk thrashing
|
|
- Larger batches (>1000): Increased memory, longer transaction locks
|
|
|
|
**Implementation**:
|
|
```python
|
|
from sqlalchemy.orm import Session
|
|
|
|
def batch_insert_captures(captures: List[Capture], batch_size=512):
|
|
session = Session()
|
|
|
|
for i in range(0, len(captures), batch_size):
|
|
batch = captures[i:i+batch_size]
|
|
session.bulk_insert_mappings(Capture, batch)
|
|
session.commit() # One commit per batch
|
|
```
|
|
|
|
---
|
|
|
|
## Decision 9: Caching - LRU Cache for Device Lookups
|
|
|
|
**Context**: Device signature matching queries database frequently for same devices.
|
|
|
|
**Decision**: Add LRU cache (64-256 entries) for device lookups.
|
|
|
|
**Rationale**:
|
|
- Wigle uses **64-entry LRU cache** for network lookups
|
|
- Reduces database queries by 90%+ (most captures match known devices)
|
|
- Small memory footprint (few KB per entry)
|
|
|
|
**Alternatives Considered**:
|
|
- No cache: Database bottleneck, slow matching
|
|
- Redis cache: Overkill for simple lookups, adds network latency
|
|
|
|
**Implementation**:
|
|
```python
|
|
from functools import lru_cache
|
|
|
|
@lru_cache(maxsize=256)
|
|
def get_device_by_id(device_id: int) -> Device:
|
|
return db.query(Device).filter_by(device_id=device_id).first()
|
|
```
|
|
|
|
---
|
|
|
|
## Decision 10: API Authentication - JWT + API Keys
|
|
|
|
**Context**: Need to authenticate users for uploads and API access.
|
|
|
|
**Decision**: Support both JWT tokens (web) and API keys (scripts/tools).
|
|
|
|
**Rationale**:
|
|
- Wigle uses **basic auth with API tokens** (retrieved via username/password)
|
|
- JWT for web sessions (short-lived, stateless)
|
|
- API keys for CLI tools (long-lived, revocable)
|
|
- Flexible for different use cases
|
|
|
|
**Alternatives Considered**:
|
|
- Username/password only: Insecure for API, no token revocation
|
|
- OAuth only: Overkill for simple use cases
|
|
|
|
**Implementation**:
|
|
```python
|
|
from fastapi import Depends, HTTPException
|
|
from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
|
|
|
|
security = HTTPBearer()
|
|
|
|
def get_current_user(credentials: HTTPAuthorizationCredentials = Depends(security)):
|
|
token = credentials.credentials
|
|
|
|
# Try JWT first
|
|
try:
|
|
payload = jwt.decode(token, SECRET_KEY, algorithms=["HS256"])
|
|
return payload["user_id"]
|
|
except:
|
|
pass
|
|
|
|
# Try API key
|
|
user = db.query(User).filter_by(api_key=token).first()
|
|
if user:
|
|
return user.user_id
|
|
|
|
raise HTTPException(status_code=401, detail="Invalid authentication")
|
|
```
|
|
|
|
---
|
|
|
|
## Summary Table
|
|
|
|
| Decision | Wigle Pattern | GigLez Adaptation | Rationale |
|
|
|----------|---------------|-------------------|-----------|
|
|
| Database | SQLite 3-table | PostgreSQL + PostGIS 3-table | Geospatial queries, ACID |
|
|
| Deduplication | BSSID primary key | SHA256 hash primary key | File-based, not MAC-based |
|
|
| Upload Format | CSV | JSON + binary | Binary file support |
|
|
| Processing | Background thread | Async task queue | Horizontal scaling |
|
|
| GPS Validation | Multi-level checks | Same checks | Proven filtering |
|
|
| Upload Marker | PREF_DB_MARKER | user_id → last_uploaded_id | Incremental sync |
|
|
| Session Tracking | run_id | session_id | Batch operations |
|
|
| Performance | Transaction batching | Same (512 ops) | Write optimization |
|
|
| Caching | 64-entry LRU | 256-entry LRU | Larger device DB |
|
|
| Auth | API tokens | JWT + API keys | Flexibility |
|
|
|
|
---
|
|
|
|
## Next Steps
|
|
|
|
1. **Implement PostgreSQL schema** (captures, devices, capture_matches, sessions)
|
|
2. **Build .sub parser** with GPS validation
|
|
3. **Create upload API** with background processing
|
|
4. **Add device matching engine** (signature database)
|
|
5. **Implement upload markers** for incremental sync
|
|
6. **Add caching layer** (LRU for devices, Redis for API responses)
|
|
7. **Build web interface** (map visualization, search, upload form)
|
|
|
|
---
|
|
|
|
**Document Version**: 1.0
|
|
**Last Updated**: 2026-01-12
|
|
**References**: /docs/wigle_analysis.md
|