giglez/docs/IMPLEMENTATION_PLAN.md

GigLez Implementation Plan - Post Wigle Analysis

Date: 2026-01-12 Status: Ready to Implement Based on: Wigle.net 15+ years of proven wardriving infrastructure

---

Executive Summary

We've completed a comprehensive analysis of Wigle.net's infrastructure, including:

• Technology stack (MySQL/MariaDB, ElasticSearch, custom mapping)
• Android client source code analysis (database schema, upload mechanisms, GPS handling)
• API documentation (authentication, CSV format, rate limiting)
• Architectural patterns (deduplication, performance optimizations, session management)

Key Finding: Wigle has successfully scaled to 349M WiFi networks and billions of observations using proven patterns we can adapt for GigLez's IoT RF device mapping.

---

Current State Assessment

✅ Completed

• Project architecture and documentation (CLAUDE.md)
• Database schema design (docs/database_schema.md)
• .sub file parser implementation (src/parser/sub_parser.py)
• Signature matching engine foundation (src/matcher/engine.py)
• Wigle infrastructure analysis (docs/wigle_analysis.md)
• Architecture decisions (docs/architecture_decisions.md)

🚧 In Progress

• None (ready to start implementation)

❌ Not Started

• PostgreSQL database setup
• SQLAlchemy ORM models
• FastAPI endpoints
• Web interface
• Signature database imports

---

Wigle Analysis Key Takeaways

1. Database Architecture

Wigle's Approach (SQLite 3-table design):

network (BSSID primary key) → Core entity storage
location (observations)      → Many-to-one with network
route (GPS tracks)           → Independent session tracking

GigLez Adaptation (PostgreSQL + PostGIS):

captures (file_hash primary key) → .sub file + GPS
capture_matches (many-to-many)   → Device identification results
devices (reference data)         → Known device signatures
sessions (run_id equivalent)     → Wardriving sessions

Why Different:

• Wigle: BSSID is a natural unique identifier (MAC address)
• GigLez: SHA256 file hash ensures deduplication of .sub files
• Wigle: Observation-level GPS (one per scan)
• GigLez: File-level GPS (one per .sub file)

2. Deduplication Strategy

Wigle's Multi-Level Approach:

1. Primary key: BSSID prevents duplicate networks
2. Upload markers: Track last uploaded ID per user
3. Spatial/temporal: Filter observations within 100m and 24h
4. Network-level: Update existing records instead of duplicating

GigLez Implementation:

# Primary deduplication (database level)
file_hash = sha256(file_contents)  # Primary key

# Spatial/temporal deduplication (optional)
existing = query_captures_within(
    latitude=lat,
    longitude=lon,
    radius_meters=50,
    time_window_hours=1
)

# Upload markers (incremental sync)
last_uploaded_id = get_user_marker(user_id, session_id)
new_captures = filter_captures_after(last_uploaded_id)

3. GPS Handling

Wigle's Quality Checks:

• Minimum accuracy threshold: 32 meters
• Null Island detection: (0.0, 0.0) rejected
• Multi-provider fallback: GPS → Network → Last known
• Location interpolation: Fill gaps between observations
• Kalman filtering: Smooth noisy GPS tracks

GigLez GPS Validation (from wigle_analysis.md):

def validate_gps(lat: float, lon: float, accuracy: float) -> bool:
    # Bounds check
    if not (-90 <= lat <= 90 and -180 <= lon <= 180):
        return False

    # Null Island check
    if abs(lat) < 0.001 and abs(lon) < 0.001:
        return False

    # Accuracy threshold
    if accuracy > 50:  # meters
        return False

    return True

4. Upload System

Wigle CSV Format:

WigleWifi-1.6,appRelease=2.70,model=Pixel,release=13,device=blueline,...
MAC,SSID,AuthMode,FirstSeen,Channel,Frequency,RSSI,CurrentLatitude,CurrentLongitude,...
00:11:22:33:44:55,MyNetwork,WPA2,2026-01-12 10:00:00,6,2437,-65,40.7128,-74.0060,...

GigLez JSON + Binary Format:

{
  "version": "GigLez-1.0",
  "session_id": "550e8400-e29b-41d4-a716-446655440000",
  "device_info": {
    "model": "LilyGo T-Embed",
    "firmware": "Bruce-2.1",
    "app_version": "1.0.0"
  },
  "captures": [
    {
      "filename": "capture_001.sub",
      "sha256": "abc123...",
      "latitude": 40.7128,
      "longitude": -74.0060,
      "accuracy": 5.0,
      "altitude": 10.5,
      "timestamp": "2026-01-12T10:00:00Z"
    }
  ]
}

5. Performance Optimizations

Wigle's Proven Patterns:

• Transaction batching (512 operations per commit)
• Prepared statements (avoid SQL parsing overhead)
• LRU caching (256 entries for device lookups)
• Background threading (offload DB writes)
• Pragma optimizations (temp_store=MEMORY, journal_mode=PERSIST)

GigLez Adaptations:

# Transaction batching (PostgreSQL)
with db.begin():
    for i in range(0, len(captures), 512):
        batch = captures[i:i+512]
        db.bulk_insert_mappings(Capture, batch)

# LRU caching (device signatures)
from functools import lru_cache

@lru_cache(maxsize=256)
def get_device_signature(device_id: int):
    return db.query(Device).filter_by(id=device_id).first()

# Async background tasks (FastAPI)
from fastapi import BackgroundTasks

@app.post("/api/captures")
async def upload_captures(files: List[UploadFile], background: BackgroundTasks):
    background.add_task(process_captures, files)
    return {"status": "processing"}

---

Implementation Roadmap

Phase 1: Foundation (Weeks 1-2)

1.1 Database Setup

Priority: CRITICAL Effort: 2 days

# Install PostgreSQL with PostGIS
sudo apt install postgresql postgresql-contrib postgis

# Create database and user
sudo -u postgres createdb giglez
sudo -u postgres createuser giglez_user -P

# Enable PostGIS extension
psql -d giglez -c "CREATE EXTENSION postgis;"

Tasks:

• Install PostgreSQL 14+ and PostGIS 3.x
• Create database and user
• Convert docs/database_schema.md to SQL script
• Add PostGIS geometry columns (ST_SetSRID)
• Create spatial indexes (GIST)
• Test basic geospatial queries

Deliverable: scripts/create_schema.sql

1.2 SQLAlchemy ORM Models

Priority: CRITICAL Effort: 3 days

# src/database/models.py
from sqlalchemy import Column, String, Integer, Float, DateTime, Text
from sqlalchemy.ext.declarative import declarative_base
from geoalchemy2 import Geometry

Base = declarative_base()

class Capture(Base):
    __tablename__ = 'captures'

    file_hash = Column(String(64), primary_key=True)
    latitude = Column(Float, nullable=False)
    longitude = Column(Float, nullable=False)
    geom = Column(Geometry('POINT', srid=4326))
    timestamp = Column(DateTime, nullable=False)
    frequency = Column(Integer, nullable=False)
    protocol = Column(String(100))
    session_id = Column(String(64))
    file_path = Column(String(500))

Tasks:

• Create ORM models for all tables
• Add relationship definitions
• Implement geom column auto-population (before_insert)
• Add validation constraints
• Create database migration with Alembic
• Write unit tests for model operations

Deliverable: src/database/models.py, alembic/versions/001_initial.py

1.3 GPS Validation Module

Priority: HIGH Effort: 1 day

Copy from docs/wigle_analysis.md Section 10 (GPS validator example).

Tasks:

• Create src/gps/validator.py
• Implement bounds checking
• Implement Null Island detection
• Implement accuracy filtering
• Add unit tests (valid/invalid cases)

Deliverable: src/gps/validator.py

---

Phase 2: Upload System (Weeks 3-4)

2.1 FastAPI Application Setup

Priority: CRITICAL Effort: 2 days

# src/api/main.py
from fastapi import FastAPI, UploadFile, File, Form
from fastapi.middleware.cors import CORSMiddleware

app = FastAPI(title="GigLez API", version="1.0.0")

app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],
    allow_methods=["*"],
    allow_headers=["*"]
)

@app.post("/api/captures/upload")
async def upload_captures(
    manifest: str = Form(...),
    files: List[UploadFile] = File(...)
):
    # Parse JSON manifest
    data = json.loads(manifest)

    # Process files
    results = await process_uploads(data, files)

    return {"uploaded": len(results), "captures": results}

Tasks:

• Create FastAPI application structure
• Add CORS middleware
• Implement authentication (JWT + API keys)
• Add rate limiting (slowapi)
• Configure logging (loguru)
• Generate OpenAPI docs

Deliverable: src/api/main.py, src/api/auth.py

2.2 File Upload Endpoint

Priority: CRITICAL Effort: 3 days

Tasks:

• Accept multipart/form-data (JSON manifest + .sub files)
• Validate manifest schema (JSON schema)
• Compute SHA256 hash per file
• Check for duplicates before processing
• Parse .sub files with existing parser
• Validate GPS coordinates
• Store files in organized directory structure
• Insert captures into database (with transaction batching)
• Return upload summary

Deliverable: src/api/routes/captures.py

2.3 Upload Marker System

Priority: MEDIUM Effort: 2 days

From docs/wigle_analysis.md Section 10:

# Track last uploaded capture ID per user/session
class UploadMarker(Base):
    __tablename__ = 'upload_markers'

    user_id = Column(Integer, primary_key=True)
    session_id = Column(String(64), primary_key=True)
    last_capture_id = Column(String(64), nullable=False)
    last_upload_time = Column(DateTime, nullable=False)

# Query for incremental sync
def get_unuploaded_captures(user_id, session_id):
    marker = get_marker(user_id, session_id)
    return query_captures_after(marker.last_capture_id)

Tasks:

• Create upload_markers table
• Implement marker update on successful upload
• Add incremental sync endpoint
• Handle resume after failed upload

Deliverable: src/database/markers.py

---

Phase 3: Device Matching (Weeks 5-6)

3.1 Signature Database Import

Priority: HIGH Effort: 4 days

Tasks:

• Clone Flipper Zero firmware repository
• Extract .sub files from assets
• Parse Flipper signatures (protocol, frequency, bit patterns)
• Import into devices and signatures tables
• Clone RTL_433 repository
• Parse protocol definitions (JSON)
• Map RTL_433 fields to database schema
• Import test data samples
• Create import scripts (idempotent)

Deliverable: scripts/import_flipper.py, scripts/import_rtl433.py

3.2 Matching Engine Implementation

Priority: HIGH Effort: 5 days

From src/matcher/engine.py (already scaffolded), implement strategies:

# src/matcher/strategies.py
class ExactMatchStrategy(MatchStrategy):
    """Exact match: protocol + frequency + bit_length"""
    def match(self, metadata, db):
        results = db.query(
            protocol=metadata.protocol,
            frequency=metadata.frequency,
            bit_length=metadata.bit_length
        )
        return [MatchResult(..., confidence=1.0) for r in results]

class PartialMatchStrategy(MatchStrategy):
    """Partial match: protocol + frequency"""
    def match(self, metadata, db):
        results = db.query(
            protocol=metadata.protocol,
            frequency=metadata.frequency
        )
        return [MatchResult(..., confidence=0.8) for r in results]

class BitPatternStrategy(MatchStrategy):
    """Bit pattern matching with masks"""
    def match(self, metadata, db):
        # Compare key_data against signature bit_patterns
        # Use bit_mask to ignore variable bits
        pass

class TimingStrategy(MatchStrategy):
    """RAW timing pattern matching"""
    def match(self, metadata, db):
        # Extract timing patterns
        # Compare against known timing signatures
        pass

Tasks:

• Implement ExactMatchStrategy
• Implement PartialMatchStrategy
• Implement BitPatternStrategy
• Implement TimingStrategy
• Add confidence scoring algorithm
• Store match results in capture_matches table
• Add unit tests for each strategy

Deliverable: src/matcher/strategies.py (complete)

---

Phase 4: Web Interface (Weeks 7-8)

4.1 Map Visualization

Priority: MEDIUM Effort: 5 days

<!-- src/web/templates/map.html -->
<!DOCTYPE html>
<html>
<head>
    <link rel="stylesheet" href="https://unpkg.com/leaflet@1.9.4/dist/leaflet.css" />
    <script src="https://unpkg.com/leaflet@1.9.4/dist/leaflet.js"></script>
</head>
<body>
    <div id="map" style="height: 600px;"></div>
    <script>
        const map = L.map('map').setView([40.7128, -74.0060], 13);

        L.tileLayer('https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png').addTo(map);

        // Fetch captures and add markers
        fetch('/api/captures?bbox=' + map.getBounds().toBBoxString())
            .then(r => r.json())
            .then(data => {
                data.captures.forEach(capture => {
                    L.marker([capture.latitude, capture.longitude])
                        .bindPopup(`
                            <b>${capture.protocol || 'Unknown'}</b><br>
                            Frequency: ${capture.frequency} Hz<br>
                            Device: ${capture.device_name || 'Unidentified'}
                        `)
                        .addTo(map);
                });
            });
    </script>
</body>
</html>

Tasks:

• Set up FastAPI static file serving
• Create Leaflet.js map interface
• Implement marker clustering (Leaflet.markercluster)
• Add heatmap layer (Leaflet.heat)
• Create device detail popup
• Add search/filter UI
• Implement bounding box query optimization

Deliverable: src/web/static/, src/web/templates/

4.2 API Query Endpoints

Priority: HIGH Effort: 3 days

@app.get("/api/captures")
async def query_captures(
    lat: float = None,
    lon: float = None,
    radius_km: float = 1.0,
    bbox: str = None,  # "min_lon,min_lat,max_lon,max_lat"
    protocol: str = None,
    device_id: int = None,
    start_time: datetime = None,
    end_time: datetime = None,
    limit: int = 100
):
    # Build geospatial query
    query = db.query(Capture)

    if lat and lon:
        # Radius query
        query = query.filter(
            func.ST_DWithin(
                Capture.geom,
                func.ST_SetSRID(func.ST_MakePoint(lon, lat), 4326),
                radius_km * 1000
            )
        )
    elif bbox:
        # Bounding box query
        min_lon, min_lat, max_lon, max_lat = map(float, bbox.split(','))
        query = query.filter(
            Capture.geom.ST_Within(
                func.ST_MakeEnvelope(min_lon, min_lat, max_lon, max_lat, 4326)
            )
        )

    # Apply filters
    if protocol:
        query = query.filter(Capture.protocol == protocol)

    results = query.limit(limit).all()
    return {"captures": [c.to_dict() for c in results]}

Tasks:

• Implement geospatial queries (radius, bounding box)
• Add filtering (protocol, frequency, device, time range)
• Implement pagination
• Add statistics endpoint (/api/stats)
• Add heatmap data endpoint (/api/heatmap)
• Add device catalog endpoint (/api/devices)
• Optimize queries with indexes

Deliverable: src/api/routes/query.py

---

Phase 5: Performance & Optimization (Weeks 9-10)

5.1 Transaction Batching

Priority: HIGH Effort: 2 days

From docs/wigle_analysis.md Section 6:

# Batch insert 512 captures per transaction
BATCH_SIZE = 512

def bulk_insert_captures(captures: List[dict]):
    with db.begin():
        for i in range(0, len(captures), BATCH_SIZE):
            batch = captures[i:i+BATCH_SIZE]
            db.bulk_insert_mappings(Capture, batch)

Tasks:

• Implement batch insert for captures
• Add batch update for existing records
• Configure SQLAlchemy connection pooling
• Add transaction retry logic
• Benchmark performance (before/after)

Deliverable: src/database/batch.py

5.2 Caching Layer

Priority: MEDIUM Effort: 3 days

from functools import lru_cache
import redis

# In-memory LRU cache for device lookups
@lru_cache(maxsize=256)
def get_device(device_id: int):
    return db.query(Device).get(device_id)

# Redis cache for API responses
redis_client = redis.Redis(host='localhost', port=6379, db=0)

@app.get("/api/devices/{device_id}")
async def get_device_api(device_id: int):
    cache_key = f"device:{device_id}"
    cached = redis_client.get(cache_key)

    if cached:
        return json.loads(cached)

    device = db.query(Device).get(device_id)
    redis_client.setex(cache_key, 3600, json.dumps(device.to_dict()))

    return device.to_dict()

Tasks:

• Add LRU caching for device/signature lookups
• Install Redis for API response caching
• Implement cache invalidation strategy
• Add cache warming on startup
• Monitor cache hit rates

Deliverable: src/cache/, Redis configuration

5.3 Materialized Views

Priority: LOW Effort: 2 days

From docs/database_schema.md (lines 362-401):

-- Pre-compute device statistics
CREATE MATERIALIZED VIEW device_statistics AS
SELECT
    d.id,
    COUNT(c.id) as total_captures,
    MIN(c.timestamp) as first_seen,
    MAX(c.timestamp) as last_seen
FROM devices d
LEFT JOIN captures c ON c.device_id = d.id
GROUP BY d.id;

-- Refresh strategy (cron job)
REFRESH MATERIALIZED VIEW CONCURRENTLY device_statistics;

Tasks:

• Create materialized views (device_statistics, geographic_heatmap)
• Set up refresh schedule (cron or pg_cron)
• Add concurrent refresh to avoid locking
• Update API to query views instead of raw tables

Deliverable: scripts/refresh_views.sh, cron configuration

---

Immediate Next Steps (This Week)

Monday-Tuesday: Database Setup

1. Install PostgreSQL and PostGIS
2. Create scripts/create_schema.sql from docs/database_schema.md
3. Add PostGIS geometry columns
4. Create spatial indexes
5. Test basic geospatial queries

Wednesday-Thursday: SQLAlchemy Models

1. Create src/database/models.py
2. Implement all table models
3. Add geom auto-population
4. Set up Alembic migrations
5. Write unit tests

Friday: GPS Validation

1. Create src/gps/validator.py
2. Implement validation functions from docs/wigle_analysis.md
3. Add unit tests
4. Integrate with upload pipeline

---

Success Metrics

Phase 1 Complete

• ✅ PostgreSQL database created with PostGIS
• ✅ All tables created from schema
• ✅ SQLAlchemy models working
• ✅ GPS validation module tested
• ✅ Can insert/query captures programmatically

MVP Complete (Phase 3)

• ✅ 1000+ signatures imported (Flipper + RTL_433)
• ✅ Automatic device matching works
• ✅ API accepts .sub file uploads
• ✅ Web interface shows captures on map
• ✅ End-to-end workflow: upload → parse → match → visualize

Production Ready (Phase 5)

• ✅ Transaction batching implemented (512 ops/commit)
• ✅ LRU caching for device lookups
• ✅ Redis caching for API responses
• ✅ Query performance < 100ms (p95)
• ✅ Can handle 10,000+ captures/day

---

Resources & References

Documentation

• docs/wigle_analysis.md - Complete Wigle client analysis
• docs/architecture_decisions.md - 10 key architectural decisions
• docs/database_schema.md - PostgreSQL schema with PostGIS
• CLAUDE.md - Project directives and requirements
• PROJECT_STATUS.md - Current status and roadmap

Code Examples

• GPS validator: docs/wigle_analysis.md Section 10
• Upload markers: docs/wigle_analysis.md Section 4
• Batch processing: docs/wigle_analysis.md Section 6
• LRU caching: docs/architecture_decisions.md Decision 9

External Resources

• Wigle Android client: https://github.com/wiglenet/wigle-wifi-wardriving
• Wigle API docs: https://api.wigle.net/
• PostGIS documentation: https://postgis.net/docs/
• FastAPI documentation: https://fastapi.tiangolo.com/
• Leaflet.js documentation: https://leafletjs.com/

---

Risk Mitigation

Technical Risks

1. Database size growth: Implement archival strategy, compress old captures
2. Query performance: Add indexes, use materialized views, implement caching
3. File storage limits: Use object storage (S3/MinIO) for .sub files
4. GPS accuracy: Follow Wigle's validation patterns (32m threshold)

Implementation Risks

1. Scope creep: Focus on MVP first (upload → parse → match → visualize)
2. Over-engineering: Start simple, optimize when needed (measure first)
3. Signature database maintenance: Automate imports with scripts

---

Next Action: Start Phase 1 database setup (PostgreSQL + PostGIS installation)