giglez/docs/architecture_decisions.md

GigLez Architecture Decisions (Based on Wigle Analysis)

Overview

This document captures key architectural decisions for GigLez, informed by analysis of the Wigle Android wardriving client.

Date: 2026-01-12 Status: Proposed Source: Wigle Analysis (/docs/wigle_analysis.md)

---

Decision 1: Database Schema - PostgreSQL with PostGIS

Context: Need to store millions of RF captures with GPS coordinates for geospatial queries.

Decision: Use PostgreSQL + PostGIS with a 3-table structure:

• captures - Primary RF file data (like Wigle's network table)
• capture_matches - Device identification results (like Wigle's location table)
• devices - Known device signature database (reference data)

Rationale:

• Wigle's 3-table design (network/location/route) has scaled to 349M WiFi networks and billions of observations
• PostGIS provides efficient geospatial indexing for radius/bounding box queries
• Separate matches table allows many-to-many relationships (one capture → multiple device matches)

Alternatives Considered:

• MongoDB: No ACID guarantees, weaker geospatial query performance
• Single table: Would duplicate device info, slower queries

Implementation:

CREATE TABLE captures (
    file_hash text primary key,
    latitude double precision not null,
    longitude double precision not null,
    geom geometry(Point, 4326),  -- PostGIS spatial index
    timestamp bigint not null,
    frequency integer not null,
    protocol text,
    session_id text
);
CREATE INDEX idx_captures_geom ON captures USING GIST(geom);

---

Decision 2: File Deduplication - SHA256 Hash as Primary Key

Context: Prevent duplicate .sub file submissions (same file uploaded multiple times).

Decision: Use SHA256 hash of file contents as primary key in captures table.

Rationale:

• Wigle uses BSSID (MAC address) as primary key for natural deduplication
• SHA256 provides cryptographically strong uniqueness guarantee
• Automatic deduplication at database level (INSERT fails on duplicate)
• No need for separate deduplication logic

Alternatives Considered:

• Auto-increment ID + separate hash check: More complex, requires application-level logic
• GPS + timestamp composite key: Could have legitimate duplicates

Implementation:

import hashlib

def compute_file_hash(file_path: str) -> str:
    sha256 = hashlib.sha256()
    with open(file_path, 'rb') as f:
        for chunk in iter(lambda: f.read(8192), b''):
            sha256.update(chunk)
    return sha256.hexdigest()

---

Decision 3: Upload Format - JSON Manifest + Binary Files

Context: Need to upload .sub files with GPS metadata to server.

Decision: Use multipart/form-data with:

• JSON manifest (GPS coordinates, metadata)
• Binary .sub files (unmodified)

Rationale:

• Wigle uses CSV for structured data export - works for text but not binary files
• JSON provides structured metadata (per-file GPS, session info)
• Binary files preserve exact capture data (no encoding issues)
• Supports batch uploads naturally (multiple files in one request)

Alternatives Considered:

• CSV with base64-encoded files: Bloated (33% overhead), parsing complexity
• Separate API calls per file: Network inefficiency, no atomic batch

Implementation:

{
  "version": "GigLez-1.0",
  "session_id": "uuid",
  "captures": [
    {
      "file": "capture_001.sub",
      "sha256": "abc123...",
      "gps": {
        "latitude": 40.7128,
        "longitude": -74.0060,
        "accuracy": 5.0,
        "timestamp": "2025-01-11T20:30:00Z"
      }
    }
  ]
}

---

Decision 4: Background Processing - Async Task Queue

Context: .sub file parsing and device matching are CPU-intensive operations.

Decision: Use async background task queue (FastAPI BackgroundTasks or Celery/RQ) for:

• File parsing
• Device signature matching
• Database writes

Rationale:

• Wigle uses background thread for all database writes (THREAD_PRIORITY_BACKGROUND)
• Prevents API timeouts on large uploads
• Enables immediate response to user (202 Accepted)
• Allows horizontal scaling of workers

Alternatives Considered:

• Synchronous processing: Slow, API timeouts on large files
• Lambda functions: Cold start overhead, cost at scale

Implementation:

from fastapi import BackgroundTasks

@app.post("/api/submit")
async def submit_capture(file: UploadFile, background_tasks: BackgroundTasks):
    file_path = await save_upload(file)
    task_id = uuid4()

    background_tasks.add_task(
        process_capture_pipeline,
        file_path=file_path,
        task_id=task_id
    )

    return {"status": "accepted", "task_id": str(task_id)}

---

Decision 5: GPS Validation - Multi-Level Checks

Context: GPS coordinates can be invalid (null island, extreme accuracy, out of bounds).

Decision: Implement Wigle's multi-level GPS validation:

1. Bounds check (-90≤lat≤90, -180≤lon≤180)
2. Accuracy threshold (reject if >100m)
3. Null island check (reject lat=0, lon=0)
4. Timestamp validation (reject if 0)

Rationale:

• Wigle's horribleGps() function filters out GPS noise from hardware bugs
• Prevents database pollution with junk coordinates
• 100m threshold balances accuracy vs. rejection rate

Alternatives Considered:

• Accept all GPS: Database pollution, poor map quality
• Strict threshold (<10m): Too restrictive, rejects valid mobile captures

Implementation:

def validate_gps(lat, lon, accuracy=None, timestamp=None):
    if not (-90 <= lat <= 90) or not (-180 <= lon <= 180):
        raise ValueError("GPS out of bounds")
    if accuracy and accuracy > 100:
        raise ValueError(f"GPS accuracy too low: {accuracy}m")
    if lat == 0.0 and lon == 0.0:
        raise ValueError("GPS at null island")
    if timestamp == 0:
        raise ValueError("Invalid timestamp")

---

Decision 6: Upload Marker System - Incremental Sync

Context: Users may upload same database multiple times (after adding new captures).

Decision: Track last uploaded capture ID per user (like Wigle's PREF_DB_MARKER).

Rationale:

• Wigle's marker system enables incremental sync (only new data)
• Prevents re-uploading entire database on each sync
• Enables resume after failure (upload only remaining captures)

Alternatives Considered:

• Always upload all captures: Wasteful bandwidth, slow
• Client-side tracking only: Lost on app reinstall

Implementation:

class UploadMarker(Base):
    __tablename__ = 'upload_markers'
    user_id = Column(String, primary_key=True)
    last_uploaded_id = Column(Integer, default=0)
    last_upload_time = Column(BigInteger)

def get_unuploaded_captures(user_id):
    marker = db.query(UploadMarker).filter_by(user_id=user_id).first()
    last_id = marker.last_uploaded_id if marker else 0

    return db.query(Capture).filter(
        Capture.id > last_id,
        Capture.uploaded == False
    ).all()

---

Decision 7: Session Management - Run ID System

Context: Group captures by wardriving session for batch operations and statistics.

Decision: Add session_id to captures table (like Wigle's run_id).

Rationale:

• Wigle's run_id groups route points by wardriving session
• Enables batch operations (upload session, view session stats)
• Useful for statistics (captures per session, coverage area)
• Supports partial uploads (session-by-session)

Alternatives Considered:

• No grouping: Hard to track wardriving runs, no batch operations
• Timestamp-based grouping: Unreliable (gaps in time)

Implementation:

CREATE TABLE sessions (
    session_id text primary key,
    user_id text not null,
    start_time bigint not null,
    end_time bigint,
    capture_count integer default 0,
    uploaded boolean default false
);

ALTER TABLE captures ADD COLUMN session_id text REFERENCES sessions(session_id);
CREATE INDEX idx_captures_session ON captures(session_id);

---

Decision 8: Performance - Transaction Batching

Context: Writing captures one-at-a-time is slow (disk I/O overhead).

Decision: Batch database writes in transactions (up to 512 operations).

Rationale:

• Wigle batches up to 512 DB updates per transaction
• Reduces disk I/O by 100x+ (one fsync per batch vs. per operation)
• Critical for write-heavy workloads (wardriving generates 1000s of observations)

Alternatives Considered:

• Individual transactions: 100x slower, disk thrashing
• Larger batches (>1000): Increased memory, longer transaction locks

Implementation:

from sqlalchemy.orm import Session

def batch_insert_captures(captures: List[Capture], batch_size=512):
    session = Session()

    for i in range(0, len(captures), batch_size):
        batch = captures[i:i+batch_size]
        session.bulk_insert_mappings(Capture, batch)
        session.commit()  # One commit per batch

---

Decision 9: Caching - LRU Cache for Device Lookups

Context: Device signature matching queries database frequently for same devices.

Decision: Add LRU cache (64-256 entries) for device lookups.

Rationale:

• Wigle uses 64-entry LRU cache for network lookups
• Reduces database queries by 90%+ (most captures match known devices)
• Small memory footprint (few KB per entry)

Alternatives Considered:

• No cache: Database bottleneck, slow matching
• Redis cache: Overkill for simple lookups, adds network latency

Implementation:

from functools import lru_cache

@lru_cache(maxsize=256)
def get_device_by_id(device_id: int) -> Device:
    return db.query(Device).filter_by(device_id=device_id).first()

---

Decision 10: API Authentication - JWT + API Keys

Context: Need to authenticate users for uploads and API access.

Decision: Support both JWT tokens (web) and API keys (scripts/tools).

Rationale:

• Wigle uses basic auth with API tokens (retrieved via username/password)
• JWT for web sessions (short-lived, stateless)
• API keys for CLI tools (long-lived, revocable)
• Flexible for different use cases

Alternatives Considered:

• Username/password only: Insecure for API, no token revocation
• OAuth only: Overkill for simple use cases

Implementation:

from fastapi import Depends, HTTPException
from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials

security = HTTPBearer()

def get_current_user(credentials: HTTPAuthorizationCredentials = Depends(security)):
    token = credentials.credentials

    # Try JWT first
    try:
        payload = jwt.decode(token, SECRET_KEY, algorithms=["HS256"])
        return payload["user_id"]
    except:
        pass

    # Try API key
    user = db.query(User).filter_by(api_key=token).first()
    if user:
        return user.user_id

    raise HTTPException(status_code=401, detail="Invalid authentication")

---

Summary Table

Decision	Wigle Pattern	GigLez Adaptation	Rationale
Database	SQLite 3-table	PostgreSQL + PostGIS 3-table	Geospatial queries, ACID
Deduplication	BSSID primary key	SHA256 hash primary key	File-based, not MAC-based
Upload Format	CSV	JSON + binary	Binary file support
Processing	Background thread	Async task queue	Horizontal scaling
GPS Validation	Multi-level checks	Same checks	Proven filtering
Upload Marker	PREF_DB_MARKER	user_id → last_uploaded_id	Incremental sync
Session Tracking	run_id	session_id	Batch operations
Performance	Transaction batching	Same (512 ops)	Write optimization
Caching	64-entry LRU	256-entry LRU	Larger device DB
Auth	API tokens	JWT + API keys	Flexibility

---

Next Steps

1. Implement PostgreSQL schema (captures, devices, capture_matches, sessions)
2. Build .sub parser with GPS validation
3. Create upload API with background processing
4. Add device matching engine (signature database)
5. Implement upload markers for incremental sync
6. Add caching layer (LRU for devices, Redis for API responses)
7. Build web interface (map visualization, search, upload form)

---

Document Version: 1.0 Last Updated: 2026-01-12 References: /docs/wigle_analysis.md