giglez/docs/IMPLEMENTATION_PLAN.md

# GigLez Implementation Plan - Post Wigle Analysis

**Date**: 2026-01-12
**Status**: Ready to Implement
**Based on**: Wigle.net 15+ years of proven wardriving infrastructure

---

## Executive Summary

We've completed a comprehensive analysis of Wigle.net's infrastructure, including:
- Technology stack (MySQL/MariaDB, ElasticSearch, custom mapping)
- Android client source code analysis (database schema, upload mechanisms, GPS handling)
- API documentation (authentication, CSV format, rate limiting)
- Architectural patterns (deduplication, performance optimizations, session management)

**Key Finding**: Wigle has successfully scaled to **349M WiFi networks** and **billions of observations** using proven patterns we can adapt for GigLez's IoT RF device mapping.

---

## Current State Assessment

### ✅ Completed
- Project architecture and documentation (CLAUDE.md)
- Database schema design (docs/database_schema.md)
- .sub file parser implementation (src/parser/sub_parser.py)
- Signature matching engine foundation (src/matcher/engine.py)
- Wigle infrastructure analysis (docs/wigle_analysis.md)
- Architecture decisions (docs/architecture_decisions.md)

### 🚧 In Progress
- None (ready to start implementation)

### ❌ Not Started
- PostgreSQL database setup
- SQLAlchemy ORM models
- FastAPI endpoints
- Web interface
- Signature database imports

---

## Wigle Analysis Key Takeaways

### 1. Database Architecture

**Wigle's Approach** (SQLite 3-table design):
```
network (BSSID primary key) → Core entity storage
location (observations)      → Many-to-one with network
route (GPS tracks)           → Independent session tracking
```

**GigLez Adaptation** (PostgreSQL + PostGIS):
```
captures (file_hash primary key) → .sub file + GPS
capture_matches (many-to-many)   → Device identification results
devices (reference data)         → Known device signatures
sessions (run_id equivalent)     → Wardriving sessions
```

**Why Different**:
- Wigle: BSSID is a natural unique identifier (MAC address)
- GigLez: SHA256 file hash ensures deduplication of .sub files
- Wigle: Observation-level GPS (one per scan)
- GigLez: File-level GPS (one per .sub file)

### 2. Deduplication Strategy

**Wigle's Multi-Level Approach**:
1. **Primary key**: BSSID prevents duplicate networks
2. **Upload markers**: Track last uploaded ID per user
3. **Spatial/temporal**: Filter observations within 100m and 24h
4. **Network-level**: Update existing records instead of duplicating

**GigLez Implementation**:
```python
# Primary deduplication (database level)
file_hash = sha256(file_contents)  # Primary key

# Spatial/temporal deduplication (optional)
existing = query_captures_within(
    latitude=lat,
    longitude=lon,
    radius_meters=50,
    time_window_hours=1
)

# Upload markers (incremental sync)
last_uploaded_id = get_user_marker(user_id, session_id)
new_captures = filter_captures_after(last_uploaded_id)
```

### 3. GPS Handling

**Wigle's Quality Checks**:
- Minimum accuracy threshold: 32 meters
- Null Island detection: (0.0, 0.0) rejected
- Multi-provider fallback: GPS → Network → Last known
- Location interpolation: Fill gaps between observations
- Kalman filtering: Smooth noisy GPS tracks

**GigLez GPS Validation** (from wigle_analysis.md):
```python
def validate_gps(lat: float, lon: float, accuracy: float) -> bool:
    # Bounds check
    if not (-90 <= lat <= 90 and -180 <= lon <= 180):
        return False

    # Null Island check
    if abs(lat) < 0.001 and abs(lon) < 0.001:
        return False

    # Accuracy threshold
    if accuracy > 50:  # meters
        return False

    return True
```

### 4. Upload System

**Wigle CSV Format**:
```csv
WigleWifi-1.6,appRelease=2.70,model=Pixel,release=13,device=blueline,...
MAC,SSID,AuthMode,FirstSeen,Channel,Frequency,RSSI,CurrentLatitude,CurrentLongitude,...
00:11:22:33:44:55,MyNetwork,WPA2,2026-01-12 10:00:00,6,2437,-65,40.7128,-74.0060,...
```

**GigLez JSON + Binary Format**:
```json
{
  "version": "GigLez-1.0",
  "session_id": "550e8400-e29b-41d4-a716-446655440000",
  "device_info": {
    "model": "LilyGo T-Embed",
    "firmware": "Bruce-2.1",
    "app_version": "1.0.0"
  },
  "captures": [
    {
      "filename": "capture_001.sub",
      "sha256": "abc123...",
      "latitude": 40.7128,
      "longitude": -74.0060,
      "accuracy": 5.0,
      "altitude": 10.5,
      "timestamp": "2026-01-12T10:00:00Z"
    }
  ]
}
```

### 5. Performance Optimizations

**Wigle's Proven Patterns**:
- Transaction batching (512 operations per commit)
- Prepared statements (avoid SQL parsing overhead)
- LRU caching (256 entries for device lookups)
- Background threading (offload DB writes)
- Pragma optimizations (temp_store=MEMORY, journal_mode=PERSIST)

**GigLez Adaptations**:
```python
# Transaction batching (PostgreSQL)
with db.begin():
    for i in range(0, len(captures), 512):
        batch = captures[i:i+512]
        db.bulk_insert_mappings(Capture, batch)

# LRU caching (device signatures)
from functools import lru_cache

@lru_cache(maxsize=256)
def get_device_signature(device_id: int):
    return db.query(Device).filter_by(id=device_id).first()

# Async background tasks (FastAPI)
from fastapi import BackgroundTasks

@app.post("/api/captures")
async def upload_captures(files: List[UploadFile], background: BackgroundTasks):
    background.add_task(process_captures, files)
    return {"status": "processing"}
```

---

## Implementation Roadmap

### Phase 1: Foundation (Weeks 1-2)

#### 1.1 Database Setup
**Priority**: CRITICAL
**Effort**: 2 days

```bash
# Install PostgreSQL with PostGIS
sudo apt install postgresql postgresql-contrib postgis

# Create database and user
sudo -u postgres createdb giglez
sudo -u postgres createuser giglez_user -P

# Enable PostGIS extension
psql -d giglez -c "CREATE EXTENSION postgis;"
```

**Tasks**:
- [ ] Install PostgreSQL 14+ and PostGIS 3.x
- [ ] Create database and user
- [ ] Convert docs/database_schema.md to SQL script
- [ ] Add PostGIS geometry columns (ST_SetSRID)
- [ ] Create spatial indexes (GIST)
- [ ] Test basic geospatial queries

**Deliverable**: `scripts/create_schema.sql`

#### 1.2 SQLAlchemy ORM Models
**Priority**: CRITICAL
**Effort**: 3 days

```python
# src/database/models.py
from sqlalchemy import Column, String, Integer, Float, DateTime, Text
from sqlalchemy.ext.declarative import declarative_base
from geoalchemy2 import Geometry

Base = declarative_base()

class Capture(Base):
    __tablename__ = 'captures'

    file_hash = Column(String(64), primary_key=True)
    latitude = Column(Float, nullable=False)
    longitude = Column(Float, nullable=False)
    geom = Column(Geometry('POINT', srid=4326))
    timestamp = Column(DateTime, nullable=False)
    frequency = Column(Integer, nullable=False)
    protocol = Column(String(100))
    session_id = Column(String(64))
    file_path = Column(String(500))
```

**Tasks**:
- [ ] Create ORM models for all tables
- [ ] Add relationship definitions
- [ ] Implement geom column auto-population (before_insert)
- [ ] Add validation constraints
- [ ] Create database migration with Alembic
- [ ] Write unit tests for model operations

**Deliverable**: `src/database/models.py`, `alembic/versions/001_initial.py`

#### 1.3 GPS Validation Module
**Priority**: HIGH
**Effort**: 1 day

Copy from `docs/wigle_analysis.md` Section 10 (GPS validator example).

**Tasks**:
- [ ] Create `src/gps/validator.py`
- [ ] Implement bounds checking
- [ ] Implement Null Island detection
- [ ] Implement accuracy filtering
- [ ] Add unit tests (valid/invalid cases)

**Deliverable**: `src/gps/validator.py`

---

### Phase 2: Upload System (Weeks 3-4)

#### 2.1 FastAPI Application Setup
**Priority**: CRITICAL
**Effort**: 2 days

```python
# src/api/main.py
from fastapi import FastAPI, UploadFile, File, Form
from fastapi.middleware.cors import CORSMiddleware

app = FastAPI(title="GigLez API", version="1.0.0")

app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],
    allow_methods=["*"],
    allow_headers=["*"]
)

@app.post("/api/captures/upload")
async def upload_captures(
    manifest: str = Form(...),
    files: List[UploadFile] = File(...)
):
    # Parse JSON manifest
    data = json.loads(manifest)

    # Process files
    results = await process_uploads(data, files)

    return {"uploaded": len(results), "captures": results}
```

**Tasks**:
- [ ] Create FastAPI application structure
- [ ] Add CORS middleware
- [ ] Implement authentication (JWT + API keys)
- [ ] Add rate limiting (slowapi)
- [ ] Configure logging (loguru)
- [ ] Generate OpenAPI docs

**Deliverable**: `src/api/main.py`, `src/api/auth.py`

#### 2.2 File Upload Endpoint
**Priority**: CRITICAL
**Effort**: 3 days

**Tasks**:
- [ ] Accept multipart/form-data (JSON manifest + .sub files)
- [ ] Validate manifest schema (JSON schema)
- [ ] Compute SHA256 hash per file
- [ ] Check for duplicates before processing
- [ ] Parse .sub files with existing parser
- [ ] Validate GPS coordinates
- [ ] Store files in organized directory structure
- [ ] Insert captures into database (with transaction batching)
- [ ] Return upload summary

**Deliverable**: `src/api/routes/captures.py`

#### 2.3 Upload Marker System
**Priority**: MEDIUM
**Effort**: 2 days

From `docs/wigle_analysis.md` Section 10:

```python
# Track last uploaded capture ID per user/session
class UploadMarker(Base):
    __tablename__ = 'upload_markers'

    user_id = Column(Integer, primary_key=True)
    session_id = Column(String(64), primary_key=True)
    last_capture_id = Column(String(64), nullable=False)
    last_upload_time = Column(DateTime, nullable=False)

# Query for incremental sync
def get_unuploaded_captures(user_id, session_id):
    marker = get_marker(user_id, session_id)
    return query_captures_after(marker.last_capture_id)
```

**Tasks**:
- [ ] Create `upload_markers` table
- [ ] Implement marker update on successful upload
- [ ] Add incremental sync endpoint
- [ ] Handle resume after failed upload

**Deliverable**: `src/database/markers.py`

---

### Phase 3: Device Matching (Weeks 5-6)

#### 3.1 Signature Database Import
**Priority**: HIGH
**Effort**: 4 days

**Tasks**:
- [ ] Clone Flipper Zero firmware repository
- [ ] Extract .sub files from assets
- [ ] Parse Flipper signatures (protocol, frequency, bit patterns)
- [ ] Import into `devices` and `signatures` tables
- [ ] Clone RTL_433 repository
- [ ] Parse protocol definitions (JSON)
- [ ] Map RTL_433 fields to database schema
- [ ] Import test data samples
- [ ] Create import scripts (idempotent)

**Deliverable**: `scripts/import_flipper.py`, `scripts/import_rtl433.py`

#### 3.2 Matching Engine Implementation
**Priority**: HIGH
**Effort**: 5 days

From `src/matcher/engine.py` (already scaffolded), implement strategies:

```python
# src/matcher/strategies.py
class ExactMatchStrategy(MatchStrategy):
    """Exact match: protocol + frequency + bit_length"""
    def match(self, metadata, db):
        results = db.query(
            protocol=metadata.protocol,
            frequency=metadata.frequency,
            bit_length=metadata.bit_length
        )
        return [MatchResult(..., confidence=1.0) for r in results]

class PartialMatchStrategy(MatchStrategy):
    """Partial match: protocol + frequency"""
    def match(self, metadata, db):
        results = db.query(
            protocol=metadata.protocol,
            frequency=metadata.frequency
        )
        return [MatchResult(..., confidence=0.8) for r in results]

class BitPatternStrategy(MatchStrategy):
    """Bit pattern matching with masks"""
    def match(self, metadata, db):
        # Compare key_data against signature bit_patterns
        # Use bit_mask to ignore variable bits
        pass

class TimingStrategy(MatchStrategy):
    """RAW timing pattern matching"""
    def match(self, metadata, db):
        # Extract timing patterns
        # Compare against known timing signatures
        pass
```

**Tasks**:
- [ ] Implement ExactMatchStrategy
- [ ] Implement PartialMatchStrategy
- [ ] Implement BitPatternStrategy
- [ ] Implement TimingStrategy
- [ ] Add confidence scoring algorithm
- [ ] Store match results in `capture_matches` table
- [ ] Add unit tests for each strategy

**Deliverable**: `src/matcher/strategies.py` (complete)

---

### Phase 4: Web Interface (Weeks 7-8)

#### 4.1 Map Visualization
**Priority**: MEDIUM
**Effort**: 5 days

```html
<!-- src/web/templates/map.html -->
<!DOCTYPE html>
<html>
<head>
    <link rel="stylesheet" href="https://unpkg.com/leaflet@1.9.4/dist/leaflet.css" />
    <script src="https://unpkg.com/leaflet@1.9.4/dist/leaflet.js"></script>
</head>
<body>
    <div id="map" style="height: 600px;"></div>
    <script>
        const map = L.map('map').setView([40.7128, -74.0060], 13);

        L.tileLayer('https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png').addTo(map);

        // Fetch captures and add markers
        fetch('/api/captures?bbox=' + map.getBounds().toBBoxString())
            .then(r => r.json())
            .then(data => {
                data.captures.forEach(capture => {
                    L.marker([capture.latitude, capture.longitude])
                        .bindPopup(`
                            <b>${capture.protocol || 'Unknown'}</b><br>
                            Frequency: ${capture.frequency} Hz<br>
                            Device: ${capture.device_name || 'Unidentified'}
                        `)
                        .addTo(map);
                });
            });
    </script>
</body>
</html>
```

**Tasks**:
- [ ] Set up FastAPI static file serving
- [ ] Create Leaflet.js map interface
- [ ] Implement marker clustering (Leaflet.markercluster)
- [ ] Add heatmap layer (Leaflet.heat)
- [ ] Create device detail popup
- [ ] Add search/filter UI
- [ ] Implement bounding box query optimization

**Deliverable**: `src/web/static/`, `src/web/templates/`

#### 4.2 API Query Endpoints
**Priority**: HIGH
**Effort**: 3 days

```python
@app.get("/api/captures")
async def query_captures(
    lat: float = None,
    lon: float = None,
    radius_km: float = 1.0,
    bbox: str = None,  # "min_lon,min_lat,max_lon,max_lat"
    protocol: str = None,
    device_id: int = None,
    start_time: datetime = None,
    end_time: datetime = None,
    limit: int = 100
):
    # Build geospatial query
    query = db.query(Capture)

    if lat and lon:
        # Radius query
        query = query.filter(
            func.ST_DWithin(
                Capture.geom,
                func.ST_SetSRID(func.ST_MakePoint(lon, lat), 4326),
                radius_km * 1000
            )
        )
    elif bbox:
        # Bounding box query
        min_lon, min_lat, max_lon, max_lat = map(float, bbox.split(','))
        query = query.filter(
            Capture.geom.ST_Within(
                func.ST_MakeEnvelope(min_lon, min_lat, max_lon, max_lat, 4326)
            )
        )

    # Apply filters
    if protocol:
        query = query.filter(Capture.protocol == protocol)

    results = query.limit(limit).all()
    return {"captures": [c.to_dict() for c in results]}
```

**Tasks**:
- [ ] Implement geospatial queries (radius, bounding box)
- [ ] Add filtering (protocol, frequency, device, time range)
- [ ] Implement pagination
- [ ] Add statistics endpoint (`/api/stats`)
- [ ] Add heatmap data endpoint (`/api/heatmap`)
- [ ] Add device catalog endpoint (`/api/devices`)
- [ ] Optimize queries with indexes

**Deliverable**: `src/api/routes/query.py`

---

### Phase 5: Performance & Optimization (Weeks 9-10)

#### 5.1 Transaction Batching
**Priority**: HIGH
**Effort**: 2 days

From `docs/wigle_analysis.md` Section 6:

```python
# Batch insert 512 captures per transaction
BATCH_SIZE = 512

def bulk_insert_captures(captures: List[dict]):
    with db.begin():
        for i in range(0, len(captures), BATCH_SIZE):
            batch = captures[i:i+BATCH_SIZE]
            db.bulk_insert_mappings(Capture, batch)
```

**Tasks**:
- [ ] Implement batch insert for captures
- [ ] Add batch update for existing records
- [ ] Configure SQLAlchemy connection pooling
- [ ] Add transaction retry logic
- [ ] Benchmark performance (before/after)

**Deliverable**: `src/database/batch.py`

#### 5.2 Caching Layer
**Priority**: MEDIUM
**Effort**: 3 days

```python
from functools import lru_cache
import redis

# In-memory LRU cache for device lookups
@lru_cache(maxsize=256)
def get_device(device_id: int):
    return db.query(Device).get(device_id)

# Redis cache for API responses
redis_client = redis.Redis(host='localhost', port=6379, db=0)

@app.get("/api/devices/{device_id}")
async def get_device_api(device_id: int):
    cache_key = f"device:{device_id}"
    cached = redis_client.get(cache_key)

    if cached:
        return json.loads(cached)

    device = db.query(Device).get(device_id)
    redis_client.setex(cache_key, 3600, json.dumps(device.to_dict()))

    return device.to_dict()
```

**Tasks**:
- [ ] Add LRU caching for device/signature lookups
- [ ] Install Redis for API response caching
- [ ] Implement cache invalidation strategy
- [ ] Add cache warming on startup
- [ ] Monitor cache hit rates

**Deliverable**: `src/cache/`, Redis configuration

#### 5.3 Materialized Views
**Priority**: LOW
**Effort**: 2 days

From `docs/database_schema.md` (lines 362-401):

```sql
-- Pre-compute device statistics
CREATE MATERIALIZED VIEW device_statistics AS
SELECT
    d.id,
    COUNT(c.id) as total_captures,
    MIN(c.timestamp) as first_seen,
    MAX(c.timestamp) as last_seen
FROM devices d
LEFT JOIN captures c ON c.device_id = d.id
GROUP BY d.id;

-- Refresh strategy (cron job)
REFRESH MATERIALIZED VIEW CONCURRENTLY device_statistics;
```

**Tasks**:
- [ ] Create materialized views (device_statistics, geographic_heatmap)
- [ ] Set up refresh schedule (cron or pg_cron)
- [ ] Add concurrent refresh to avoid locking
- [ ] Update API to query views instead of raw tables

**Deliverable**: `scripts/refresh_views.sh`, cron configuration

---

## Immediate Next Steps (This Week)

### Monday-Tuesday: Database Setup
1. Install PostgreSQL and PostGIS
2. Create `scripts/create_schema.sql` from `docs/database_schema.md`
3. Add PostGIS geometry columns
4. Create spatial indexes
5. Test basic geospatial queries

### Wednesday-Thursday: SQLAlchemy Models
1. Create `src/database/models.py`
2. Implement all table models
3. Add geom auto-population
4. Set up Alembic migrations
5. Write unit tests

### Friday: GPS Validation
1. Create `src/gps/validator.py`
2. Implement validation functions from `docs/wigle_analysis.md`
3. Add unit tests
4. Integrate with upload pipeline

---

## Success Metrics

### Phase 1 Complete
- ✅ PostgreSQL database created with PostGIS
- ✅ All tables created from schema
- ✅ SQLAlchemy models working
- ✅ GPS validation module tested
- ✅ Can insert/query captures programmatically

### MVP Complete (Phase 3)
- ✅ 1000+ signatures imported (Flipper + RTL_433)
- ✅ Automatic device matching works
- ✅ API accepts .sub file uploads
- ✅ Web interface shows captures on map
- ✅ End-to-end workflow: upload → parse → match → visualize

### Production Ready (Phase 5)
- ✅ Transaction batching implemented (512 ops/commit)
- ✅ LRU caching for device lookups
- ✅ Redis caching for API responses
- ✅ Query performance < 100ms (p95)
- ✅ Can handle 10,000+ captures/day

---

## Resources & References

### Documentation
- `docs/wigle_analysis.md` - Complete Wigle client analysis
- `docs/architecture_decisions.md` - 10 key architectural decisions
- `docs/database_schema.md` - PostgreSQL schema with PostGIS
- `CLAUDE.md` - Project directives and requirements
- `PROJECT_STATUS.md` - Current status and roadmap

### Code Examples
- GPS validator: `docs/wigle_analysis.md` Section 10
- Upload markers: `docs/wigle_analysis.md` Section 4
- Batch processing: `docs/wigle_analysis.md` Section 6
- LRU caching: `docs/architecture_decisions.md` Decision 9

### External Resources
- Wigle Android client: https://github.com/wiglenet/wigle-wifi-wardriving
- Wigle API docs: https://api.wigle.net/
- PostGIS documentation: https://postgis.net/docs/
- FastAPI documentation: https://fastapi.tiangolo.com/
- Leaflet.js documentation: https://leafletjs.com/

---

## Risk Mitigation

### Technical Risks
1. **Database size growth**: Implement archival strategy, compress old captures
2. **Query performance**: Add indexes, use materialized views, implement caching
3. **File storage limits**: Use object storage (S3/MinIO) for .sub files
4. **GPS accuracy**: Follow Wigle's validation patterns (32m threshold)

### Implementation Risks
1. **Scope creep**: Focus on MVP first (upload → parse → match → visualize)
2. **Over-engineering**: Start simple, optimize when needed (measure first)
3. **Signature database maintenance**: Automate imports with scripts

---

**Next Action**: Start Phase 1 database setup (PostgreSQL + PostGIS installation)