# GigLez Architecture Decisions (Based on Wigle Analysis)

## Overview

This document captures key architectural decisions for GigLez, informed by analysis of the Wigle Android wardriving client.

**Date**: 2026-01-12
**Status**: Proposed
**Source**: Wigle Analysis (/docs/wigle_analysis.md)

---

## Decision 1: Database Schema - PostgreSQL with PostGIS

**Context**: Need to store millions of RF captures with GPS coordinates for geospatial queries.

**Decision**: Use PostgreSQL + PostGIS with a 3-table structure:
- `captures` - Primary RF file data (like Wigle's `network` table)
- `capture_matches` - Device identification results (like Wigle's `location` table)
- `devices` - Known device signature database (reference data)

**Rationale**:
- Wigle's 3-table design (network/location/route) has scaled to **349M WiFi networks** and **billions of observations**
- PostGIS provides efficient geospatial indexing for radius/bounding box queries
- Separate matches table allows many-to-many relationships (one capture → multiple device matches)

**Alternatives Considered**:
- MongoDB: No ACID guarantees, weaker geospatial query performance
- Single table: Would duplicate device info, slower queries

**Implementation**:
```sql
CREATE TABLE captures (
    file_hash text primary key,
    latitude double precision not null,
    longitude double precision not null,
    geom geometry(Point, 4326),  -- PostGIS spatial index
    timestamp bigint not null,
    frequency integer not null,
    protocol text,
    session_id text
);
CREATE INDEX idx_captures_geom ON captures USING GIST(geom);
```

---

## Decision 2: File Deduplication - SHA256 Hash as Primary Key

**Context**: Prevent duplicate .sub file submissions (same file uploaded multiple times).

**Decision**: Use SHA256 hash of file contents as primary key in `captures` table.

**Rationale**:
- Wigle uses BSSID (MAC address) as primary key for natural deduplication
- SHA256 provides **cryptographically strong** uniqueness guarantee
- Automatic deduplication at database level (INSERT fails on duplicate)
- No need for separate deduplication logic

**Alternatives Considered**:
- Auto-increment ID + separate hash check: More complex, requires application-level logic
- GPS + timestamp composite key: Could have legitimate duplicates

**Implementation**:
```python
import hashlib

def compute_file_hash(file_path: str) -> str:
    sha256 = hashlib.sha256()
    with open(file_path, 'rb') as f:
        for chunk in iter(lambda: f.read(8192), b''):
            sha256.update(chunk)
    return sha256.hexdigest()
```

---

## Decision 3: Upload Format - JSON Manifest + Binary Files

**Context**: Need to upload .sub files with GPS metadata to server.

**Decision**: Use multipart/form-data with:
- JSON manifest (GPS coordinates, metadata)
- Binary .sub files (unmodified)

**Rationale**:
- Wigle uses CSV for structured data export - works for text but not binary files
- JSON provides structured metadata (per-file GPS, session info)
- Binary files preserve exact capture data (no encoding issues)
- Supports batch uploads naturally (multiple files in one request)

**Alternatives Considered**:
- CSV with base64-encoded files: Bloated (33% overhead), parsing complexity
- Separate API calls per file: Network inefficiency, no atomic batch

**Implementation**:
```json
{
  "version": "GigLez-1.0",
  "session_id": "uuid",
  "captures": [
    {
      "file": "capture_001.sub",
      "sha256": "abc123...",
      "gps": {
        "latitude": 40.7128,
        "longitude": -74.0060,
        "accuracy": 5.0,
        "timestamp": "2025-01-11T20:30:00Z"
      }
    }
  ]
}
```

---

## Decision 4: Background Processing - Async Task Queue

**Context**: .sub file parsing and device matching are CPU-intensive operations.

**Decision**: Use async background task queue (FastAPI BackgroundTasks or Celery/RQ) for:
- File parsing
- Device signature matching
- Database writes

**Rationale**:
- Wigle uses **background thread** for all database writes (THREAD_PRIORITY_BACKGROUND)
- Prevents API timeouts on large uploads
- Enables **immediate response** to user (202 Accepted)
- Allows **horizontal scaling** of workers

**Alternatives Considered**:
- Synchronous processing: Slow, API timeouts on large files
- Lambda functions: Cold start overhead, cost at scale

**Implementation**:
```python
from fastapi import BackgroundTasks

@app.post("/api/submit")
async def submit_capture(file: UploadFile, background_tasks: BackgroundTasks):
    file_path = await save_upload(file)
    task_id = uuid4()

    background_tasks.add_task(
        process_capture_pipeline,
        file_path=file_path,
        task_id=task_id
    )

    return {"status": "accepted", "task_id": str(task_id)}
```

---

## Decision 5: GPS Validation - Multi-Level Checks

**Context**: GPS coordinates can be invalid (null island, extreme accuracy, out of bounds).

**Decision**: Implement Wigle's multi-level GPS validation:
1. Bounds check (-90≤lat≤90, -180≤lon≤180)
2. Accuracy threshold (reject if >100m)
3. Null island check (reject lat=0, lon=0)
4. Timestamp validation (reject if 0)

**Rationale**:
- Wigle's horribleGps() function filters out **GPS noise** from hardware bugs
- Prevents database pollution with junk coordinates
- 100m threshold balances accuracy vs. rejection rate

**Alternatives Considered**:
- Accept all GPS: Database pollution, poor map quality
- Strict threshold (<10m): Too restrictive, rejects valid mobile captures

**Implementation**:
```python
def validate_gps(lat, lon, accuracy=None, timestamp=None):
    if not (-90 <= lat <= 90) or not (-180 <= lon <= 180):
        raise ValueError("GPS out of bounds")
    if accuracy and accuracy > 100:
        raise ValueError(f"GPS accuracy too low: {accuracy}m")
    if lat == 0.0 and lon == 0.0:
        raise ValueError("GPS at null island")
    if timestamp == 0:
        raise ValueError("Invalid timestamp")
```

---

## Decision 6: Upload Marker System - Incremental Sync

**Context**: Users may upload same database multiple times (after adding new captures).

**Decision**: Track last uploaded capture ID per user (like Wigle's PREF_DB_MARKER).

**Rationale**:
- Wigle's marker system enables **incremental sync** (only new data)
- Prevents re-uploading entire database on each sync
- Enables **resume after failure** (upload only remaining captures)

**Alternatives Considered**:
- Always upload all captures: Wasteful bandwidth, slow
- Client-side tracking only: Lost on app reinstall

**Implementation**:
```python
class UploadMarker(Base):
    __tablename__ = 'upload_markers'
    user_id = Column(String, primary_key=True)
    last_uploaded_id = Column(Integer, default=0)
    last_upload_time = Column(BigInteger)

def get_unuploaded_captures(user_id):
    marker = db.query(UploadMarker).filter_by(user_id=user_id).first()
    last_id = marker.last_uploaded_id if marker else 0

    return db.query(Capture).filter(
        Capture.id > last_id,
        Capture.uploaded == False
    ).all()
```

---

## Decision 7: Session Management - Run ID System

**Context**: Group captures by wardriving session for batch operations and statistics.

**Decision**: Add `session_id` to captures table (like Wigle's `run_id`).

**Rationale**:
- Wigle's run_id groups **route points** by wardriving session
- Enables batch operations (upload session, view session stats)
- Useful for statistics (captures per session, coverage area)
- Supports partial uploads (session-by-session)

**Alternatives Considered**:
- No grouping: Hard to track wardriving runs, no batch operations
- Timestamp-based grouping: Unreliable (gaps in time)

**Implementation**:
```sql
CREATE TABLE sessions (
    session_id text primary key,
    user_id text not null,
    start_time bigint not null,
    end_time bigint,
    capture_count integer default 0,
    uploaded boolean default false
);

ALTER TABLE captures ADD COLUMN session_id text REFERENCES sessions(session_id);
CREATE INDEX idx_captures_session ON captures(session_id);
```

---

## Decision 8: Performance - Transaction Batching

**Context**: Writing captures one-at-a-time is slow (disk I/O overhead).

**Decision**: Batch database writes in transactions (up to 512 operations).

**Rationale**:
- Wigle batches up to **512 DB updates** per transaction
- Reduces disk I/O by 100x+ (one fsync per batch vs. per operation)
- Critical for write-heavy workloads (wardriving generates 1000s of observations)

**Alternatives Considered**:
- Individual transactions: 100x slower, disk thrashing
- Larger batches (>1000): Increased memory, longer transaction locks

**Implementation**:
```python
from sqlalchemy.orm import Session

def batch_insert_captures(captures: List[Capture], batch_size=512):
    session = Session()

    for i in range(0, len(captures), batch_size):
        batch = captures[i:i+batch_size]
        session.bulk_insert_mappings(Capture, batch)
        session.commit()  # One commit per batch
```

---

## Decision 9: Caching - LRU Cache for Device Lookups

**Context**: Device signature matching queries database frequently for same devices.

**Decision**: Add LRU cache (64-256 entries) for device lookups.

**Rationale**:
- Wigle uses **64-entry LRU cache** for network lookups
- Reduces database queries by 90%+ (most captures match known devices)
- Small memory footprint (few KB per entry)

**Alternatives Considered**:
- No cache: Database bottleneck, slow matching
- Redis cache: Overkill for simple lookups, adds network latency

**Implementation**:
```python
from functools import lru_cache

@lru_cache(maxsize=256)
def get_device_by_id(device_id: int) -> Device:
    return db.query(Device).filter_by(device_id=device_id).first()
```

---

## Decision 10: API Authentication - JWT + API Keys

**Context**: Need to authenticate users for uploads and API access.

**Decision**: Support both JWT tokens (web) and API keys (scripts/tools).

**Rationale**:
- Wigle uses **basic auth with API tokens** (retrieved via username/password)
- JWT for web sessions (short-lived, stateless)
- API keys for CLI tools (long-lived, revocable)
- Flexible for different use cases

**Alternatives Considered**:
- Username/password only: Insecure for API, no token revocation
- OAuth only: Overkill for simple use cases

**Implementation**:
```python
from fastapi import Depends, HTTPException
from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials

security = HTTPBearer()

def get_current_user(credentials: HTTPAuthorizationCredentials = Depends(security)):
    token = credentials.credentials

    # Try JWT first
    try:
        payload = jwt.decode(token, SECRET_KEY, algorithms=["HS256"])
        return payload["user_id"]
    except:
        pass

    # Try API key
    user = db.query(User).filter_by(api_key=token).first()
    if user:
        return user.user_id

    raise HTTPException(status_code=401, detail="Invalid authentication")
```

---

## Summary Table

| Decision | Wigle Pattern | GigLez Adaptation | Rationale |
|----------|---------------|-------------------|-----------|
| Database | SQLite 3-table | PostgreSQL + PostGIS 3-table | Geospatial queries, ACID |
| Deduplication | BSSID primary key | SHA256 hash primary key | File-based, not MAC-based |
| Upload Format | CSV | JSON + binary | Binary file support |
| Processing | Background thread | Async task queue | Horizontal scaling |
| GPS Validation | Multi-level checks | Same checks | Proven filtering |
| Upload Marker | PREF_DB_MARKER | user_id → last_uploaded_id | Incremental sync |
| Session Tracking | run_id | session_id | Batch operations |
| Performance | Transaction batching | Same (512 ops) | Write optimization |
| Caching | 64-entry LRU | 256-entry LRU | Larger device DB |
| Auth | API tokens | JWT + API keys | Flexibility |

---

## Next Steps

1. **Implement PostgreSQL schema** (captures, devices, capture_matches, sessions)
2. **Build .sub parser** with GPS validation
3. **Create upload API** with background processing
4. **Add device matching engine** (signature database)
5. **Implement upload markers** for incremental sync
6. **Add caching layer** (LRU for devices, Redis for API responses)
7. **Build web interface** (map visualization, search, upload form)

---

**Document Version**: 1.0
**Last Updated**: 2026-01-12
**References**: /docs/wigle_analysis.md