Initial commit: Phase 1 & Phase 2 infrastructure complete
This commit is contained in:
@@ -0,0 +1,389 @@
|
||||
# GigLez Architecture Decisions (Based on Wigle Analysis)
|
||||
|
||||
## Overview
|
||||
|
||||
This document captures key architectural decisions for GigLez, informed by analysis of the Wigle Android wardriving client.
|
||||
|
||||
**Date**: 2026-01-12
|
||||
**Status**: Proposed
|
||||
**Source**: Wigle Analysis (/docs/wigle_analysis.md)
|
||||
|
||||
---
|
||||
|
||||
## Decision 1: Database Schema - PostgreSQL with PostGIS
|
||||
|
||||
**Context**: Need to store millions of RF captures with GPS coordinates for geospatial queries.
|
||||
|
||||
**Decision**: Use PostgreSQL + PostGIS with a 3-table structure:
|
||||
- `captures` - Primary RF file data (like Wigle's `network` table)
|
||||
- `capture_matches` - Device identification results (like Wigle's `location` table)
|
||||
- `devices` - Known device signature database (reference data)
|
||||
|
||||
**Rationale**:
|
||||
- Wigle's 3-table design (network/location/route) has scaled to **349M WiFi networks** and **billions of observations**
|
||||
- PostGIS provides efficient geospatial indexing for radius/bounding box queries
|
||||
- Separate matches table allows many-to-many relationships (one capture → multiple device matches)
|
||||
|
||||
**Alternatives Considered**:
|
||||
- MongoDB: No ACID guarantees, weaker geospatial query performance
|
||||
- Single table: Would duplicate device info, slower queries
|
||||
|
||||
**Implementation**:
|
||||
```sql
|
||||
CREATE TABLE captures (
|
||||
file_hash text primary key,
|
||||
latitude double precision not null,
|
||||
longitude double precision not null,
|
||||
geom geometry(Point, 4326), -- PostGIS spatial index
|
||||
timestamp bigint not null,
|
||||
frequency integer not null,
|
||||
protocol text,
|
||||
session_id text
|
||||
);
|
||||
CREATE INDEX idx_captures_geom ON captures USING GIST(geom);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Decision 2: File Deduplication - SHA256 Hash as Primary Key
|
||||
|
||||
**Context**: Prevent duplicate .sub file submissions (same file uploaded multiple times).
|
||||
|
||||
**Decision**: Use SHA256 hash of file contents as primary key in `captures` table.
|
||||
|
||||
**Rationale**:
|
||||
- Wigle uses BSSID (MAC address) as primary key for natural deduplication
|
||||
- SHA256 provides **cryptographically strong** uniqueness guarantee
|
||||
- Automatic deduplication at database level (INSERT fails on duplicate)
|
||||
- No need for separate deduplication logic
|
||||
|
||||
**Alternatives Considered**:
|
||||
- Auto-increment ID + separate hash check: More complex, requires application-level logic
|
||||
- GPS + timestamp composite key: Could have legitimate duplicates
|
||||
|
||||
**Implementation**:
|
||||
```python
|
||||
import hashlib
|
||||
|
||||
def compute_file_hash(file_path: str) -> str:
|
||||
sha256 = hashlib.sha256()
|
||||
with open(file_path, 'rb') as f:
|
||||
for chunk in iter(lambda: f.read(8192), b''):
|
||||
sha256.update(chunk)
|
||||
return sha256.hexdigest()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Decision 3: Upload Format - JSON Manifest + Binary Files
|
||||
|
||||
**Context**: Need to upload .sub files with GPS metadata to server.
|
||||
|
||||
**Decision**: Use multipart/form-data with:
|
||||
- JSON manifest (GPS coordinates, metadata)
|
||||
- Binary .sub files (unmodified)
|
||||
|
||||
**Rationale**:
|
||||
- Wigle uses CSV for structured data export - works for text but not binary files
|
||||
- JSON provides structured metadata (per-file GPS, session info)
|
||||
- Binary files preserve exact capture data (no encoding issues)
|
||||
- Supports batch uploads naturally (multiple files in one request)
|
||||
|
||||
**Alternatives Considered**:
|
||||
- CSV with base64-encoded files: Bloated (33% overhead), parsing complexity
|
||||
- Separate API calls per file: Network inefficiency, no atomic batch
|
||||
|
||||
**Implementation**:
|
||||
```json
|
||||
{
|
||||
"version": "GigLez-1.0",
|
||||
"session_id": "uuid",
|
||||
"captures": [
|
||||
{
|
||||
"file": "capture_001.sub",
|
||||
"sha256": "abc123...",
|
||||
"gps": {
|
||||
"latitude": 40.7128,
|
||||
"longitude": -74.0060,
|
||||
"accuracy": 5.0,
|
||||
"timestamp": "2025-01-11T20:30:00Z"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Decision 4: Background Processing - Async Task Queue
|
||||
|
||||
**Context**: .sub file parsing and device matching are CPU-intensive operations.
|
||||
|
||||
**Decision**: Use async background task queue (FastAPI BackgroundTasks or Celery/RQ) for:
|
||||
- File parsing
|
||||
- Device signature matching
|
||||
- Database writes
|
||||
|
||||
**Rationale**:
|
||||
- Wigle uses **background thread** for all database writes (THREAD_PRIORITY_BACKGROUND)
|
||||
- Prevents API timeouts on large uploads
|
||||
- Enables **immediate response** to user (202 Accepted)
|
||||
- Allows **horizontal scaling** of workers
|
||||
|
||||
**Alternatives Considered**:
|
||||
- Synchronous processing: Slow, API timeouts on large files
|
||||
- Lambda functions: Cold start overhead, cost at scale
|
||||
|
||||
**Implementation**:
|
||||
```python
|
||||
from fastapi import BackgroundTasks
|
||||
|
||||
@app.post("/api/submit")
|
||||
async def submit_capture(file: UploadFile, background_tasks: BackgroundTasks):
|
||||
file_path = await save_upload(file)
|
||||
task_id = uuid4()
|
||||
|
||||
background_tasks.add_task(
|
||||
process_capture_pipeline,
|
||||
file_path=file_path,
|
||||
task_id=task_id
|
||||
)
|
||||
|
||||
return {"status": "accepted", "task_id": str(task_id)}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Decision 5: GPS Validation - Multi-Level Checks
|
||||
|
||||
**Context**: GPS coordinates can be invalid (null island, extreme accuracy, out of bounds).
|
||||
|
||||
**Decision**: Implement Wigle's multi-level GPS validation:
|
||||
1. Bounds check (-90≤lat≤90, -180≤lon≤180)
|
||||
2. Accuracy threshold (reject if >100m)
|
||||
3. Null island check (reject lat=0, lon=0)
|
||||
4. Timestamp validation (reject if 0)
|
||||
|
||||
**Rationale**:
|
||||
- Wigle's horribleGps() function filters out **GPS noise** from hardware bugs
|
||||
- Prevents database pollution with junk coordinates
|
||||
- 100m threshold balances accuracy vs. rejection rate
|
||||
|
||||
**Alternatives Considered**:
|
||||
- Accept all GPS: Database pollution, poor map quality
|
||||
- Strict threshold (<10m): Too restrictive, rejects valid mobile captures
|
||||
|
||||
**Implementation**:
|
||||
```python
|
||||
def validate_gps(lat, lon, accuracy=None, timestamp=None):
|
||||
if not (-90 <= lat <= 90) or not (-180 <= lon <= 180):
|
||||
raise ValueError("GPS out of bounds")
|
||||
if accuracy and accuracy > 100:
|
||||
raise ValueError(f"GPS accuracy too low: {accuracy}m")
|
||||
if lat == 0.0 and lon == 0.0:
|
||||
raise ValueError("GPS at null island")
|
||||
if timestamp == 0:
|
||||
raise ValueError("Invalid timestamp")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Decision 6: Upload Marker System - Incremental Sync
|
||||
|
||||
**Context**: Users may upload same database multiple times (after adding new captures).
|
||||
|
||||
**Decision**: Track last uploaded capture ID per user (like Wigle's PREF_DB_MARKER).
|
||||
|
||||
**Rationale**:
|
||||
- Wigle's marker system enables **incremental sync** (only new data)
|
||||
- Prevents re-uploading entire database on each sync
|
||||
- Enables **resume after failure** (upload only remaining captures)
|
||||
|
||||
**Alternatives Considered**:
|
||||
- Always upload all captures: Wasteful bandwidth, slow
|
||||
- Client-side tracking only: Lost on app reinstall
|
||||
|
||||
**Implementation**:
|
||||
```python
|
||||
class UploadMarker(Base):
|
||||
__tablename__ = 'upload_markers'
|
||||
user_id = Column(String, primary_key=True)
|
||||
last_uploaded_id = Column(Integer, default=0)
|
||||
last_upload_time = Column(BigInteger)
|
||||
|
||||
def get_unuploaded_captures(user_id):
|
||||
marker = db.query(UploadMarker).filter_by(user_id=user_id).first()
|
||||
last_id = marker.last_uploaded_id if marker else 0
|
||||
|
||||
return db.query(Capture).filter(
|
||||
Capture.id > last_id,
|
||||
Capture.uploaded == False
|
||||
).all()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Decision 7: Session Management - Run ID System
|
||||
|
||||
**Context**: Group captures by wardriving session for batch operations and statistics.
|
||||
|
||||
**Decision**: Add `session_id` to captures table (like Wigle's `run_id`).
|
||||
|
||||
**Rationale**:
|
||||
- Wigle's run_id groups **route points** by wardriving session
|
||||
- Enables batch operations (upload session, view session stats)
|
||||
- Useful for statistics (captures per session, coverage area)
|
||||
- Supports partial uploads (session-by-session)
|
||||
|
||||
**Alternatives Considered**:
|
||||
- No grouping: Hard to track wardriving runs, no batch operations
|
||||
- Timestamp-based grouping: Unreliable (gaps in time)
|
||||
|
||||
**Implementation**:
|
||||
```sql
|
||||
CREATE TABLE sessions (
|
||||
session_id text primary key,
|
||||
user_id text not null,
|
||||
start_time bigint not null,
|
||||
end_time bigint,
|
||||
capture_count integer default 0,
|
||||
uploaded boolean default false
|
||||
);
|
||||
|
||||
ALTER TABLE captures ADD COLUMN session_id text REFERENCES sessions(session_id);
|
||||
CREATE INDEX idx_captures_session ON captures(session_id);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Decision 8: Performance - Transaction Batching
|
||||
|
||||
**Context**: Writing captures one-at-a-time is slow (disk I/O overhead).
|
||||
|
||||
**Decision**: Batch database writes in transactions (up to 512 operations).
|
||||
|
||||
**Rationale**:
|
||||
- Wigle batches up to **512 DB updates** per transaction
|
||||
- Reduces disk I/O by 100x+ (one fsync per batch vs. per operation)
|
||||
- Critical for write-heavy workloads (wardriving generates 1000s of observations)
|
||||
|
||||
**Alternatives Considered**:
|
||||
- Individual transactions: 100x slower, disk thrashing
|
||||
- Larger batches (>1000): Increased memory, longer transaction locks
|
||||
|
||||
**Implementation**:
|
||||
```python
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
def batch_insert_captures(captures: List[Capture], batch_size=512):
|
||||
session = Session()
|
||||
|
||||
for i in range(0, len(captures), batch_size):
|
||||
batch = captures[i:i+batch_size]
|
||||
session.bulk_insert_mappings(Capture, batch)
|
||||
session.commit() # One commit per batch
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Decision 9: Caching - LRU Cache for Device Lookups
|
||||
|
||||
**Context**: Device signature matching queries database frequently for same devices.
|
||||
|
||||
**Decision**: Add LRU cache (64-256 entries) for device lookups.
|
||||
|
||||
**Rationale**:
|
||||
- Wigle uses **64-entry LRU cache** for network lookups
|
||||
- Reduces database queries by 90%+ (most captures match known devices)
|
||||
- Small memory footprint (few KB per entry)
|
||||
|
||||
**Alternatives Considered**:
|
||||
- No cache: Database bottleneck, slow matching
|
||||
- Redis cache: Overkill for simple lookups, adds network latency
|
||||
|
||||
**Implementation**:
|
||||
```python
|
||||
from functools import lru_cache
|
||||
|
||||
@lru_cache(maxsize=256)
|
||||
def get_device_by_id(device_id: int) -> Device:
|
||||
return db.query(Device).filter_by(device_id=device_id).first()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Decision 10: API Authentication - JWT + API Keys
|
||||
|
||||
**Context**: Need to authenticate users for uploads and API access.
|
||||
|
||||
**Decision**: Support both JWT tokens (web) and API keys (scripts/tools).
|
||||
|
||||
**Rationale**:
|
||||
- Wigle uses **basic auth with API tokens** (retrieved via username/password)
|
||||
- JWT for web sessions (short-lived, stateless)
|
||||
- API keys for CLI tools (long-lived, revocable)
|
||||
- Flexible for different use cases
|
||||
|
||||
**Alternatives Considered**:
|
||||
- Username/password only: Insecure for API, no token revocation
|
||||
- OAuth only: Overkill for simple use cases
|
||||
|
||||
**Implementation**:
|
||||
```python
|
||||
from fastapi import Depends, HTTPException
|
||||
from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
|
||||
|
||||
security = HTTPBearer()
|
||||
|
||||
def get_current_user(credentials: HTTPAuthorizationCredentials = Depends(security)):
|
||||
token = credentials.credentials
|
||||
|
||||
# Try JWT first
|
||||
try:
|
||||
payload = jwt.decode(token, SECRET_KEY, algorithms=["HS256"])
|
||||
return payload["user_id"]
|
||||
except:
|
||||
pass
|
||||
|
||||
# Try API key
|
||||
user = db.query(User).filter_by(api_key=token).first()
|
||||
if user:
|
||||
return user.user_id
|
||||
|
||||
raise HTTPException(status_code=401, detail="Invalid authentication")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Summary Table
|
||||
|
||||
| Decision | Wigle Pattern | GigLez Adaptation | Rationale |
|
||||
|----------|---------------|-------------------|-----------|
|
||||
| Database | SQLite 3-table | PostgreSQL + PostGIS 3-table | Geospatial queries, ACID |
|
||||
| Deduplication | BSSID primary key | SHA256 hash primary key | File-based, not MAC-based |
|
||||
| Upload Format | CSV | JSON + binary | Binary file support |
|
||||
| Processing | Background thread | Async task queue | Horizontal scaling |
|
||||
| GPS Validation | Multi-level checks | Same checks | Proven filtering |
|
||||
| Upload Marker | PREF_DB_MARKER | user_id → last_uploaded_id | Incremental sync |
|
||||
| Session Tracking | run_id | session_id | Batch operations |
|
||||
| Performance | Transaction batching | Same (512 ops) | Write optimization |
|
||||
| Caching | 64-entry LRU | 256-entry LRU | Larger device DB |
|
||||
| Auth | API tokens | JWT + API keys | Flexibility |
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Implement PostgreSQL schema** (captures, devices, capture_matches, sessions)
|
||||
2. **Build .sub parser** with GPS validation
|
||||
3. **Create upload API** with background processing
|
||||
4. **Add device matching engine** (signature database)
|
||||
5. **Implement upload markers** for incremental sync
|
||||
6. **Add caching layer** (LRU for devices, Redis for API responses)
|
||||
7. **Build web interface** (map visualization, search, upload form)
|
||||
|
||||
---
|
||||
|
||||
**Document Version**: 1.0
|
||||
**Last Updated**: 2026-01-12
|
||||
**References**: /docs/wigle_analysis.md
|
||||
Reference in New Issue
Block a user