Files

T

Trilltechnician eb225771bc Initial commit: Phase 1 & Phase 2 infrastructure complete

2026-01-12 11:21:17 -08:00

20 KiB

Raw Blame History

GigLez Implementation Plan - Post Wigle Analysis

Date: 2026-01-12 Status: Ready to Implement Based on: Wigle.net 15+ years of proven wardriving infrastructure

Executive Summary

We've completed a comprehensive analysis of Wigle.net's infrastructure, including:

Technology stack (MySQL/MariaDB, ElasticSearch, custom mapping)
Android client source code analysis (database schema, upload mechanisms, GPS handling)
API documentation (authentication, CSV format, rate limiting)
Architectural patterns (deduplication, performance optimizations, session management)

Key Finding: Wigle has successfully scaled to 349M WiFi networks and billions of observations using proven patterns we can adapt for GigLez's IoT RF device mapping.

Current State Assessment

✅ Completed

Project architecture and documentation (CLAUDE.md)
Database schema design (docs/database_schema.md)
.sub file parser implementation (src/parser/sub_parser.py)
Signature matching engine foundation (src/matcher/engine.py)
Wigle infrastructure analysis (docs/wigle_analysis.md)
Architecture decisions (docs/architecture_decisions.md)

🚧 In Progress

None (ready to start implementation)

❌ Not Started

PostgreSQL database setup
SQLAlchemy ORM models
FastAPI endpoints
Web interface
Signature database imports

Wigle Analysis Key Takeaways

1. Database Architecture

Wigle's Approach (SQLite 3-table design):

network (BSSID primary key) → Core entity storage
location (observations)      → Many-to-one with network
route (GPS tracks)           → Independent session tracking

GigLez Adaptation (PostgreSQL + PostGIS):

captures (file_hash primary key) → .sub file + GPS
capture_matches (many-to-many)   → Device identification results
devices (reference data)         → Known device signatures
sessions (run_id equivalent)     → Wardriving sessions

Why Different:

Wigle: BSSID is a natural unique identifier (MAC address)
GigLez: SHA256 file hash ensures deduplication of .sub files
Wigle: Observation-level GPS (one per scan)
GigLez: File-level GPS (one per .sub file)

2. Deduplication Strategy

Wigle's Multi-Level Approach:

Primary key: BSSID prevents duplicate networks
Upload markers: Track last uploaded ID per user
Spatial/temporal: Filter observations within 100m and 24h
Network-level: Update existing records instead of duplicating

GigLez Implementation:

# Primary deduplication (database level)
file_hash = sha256(file_contents)  # Primary key

# Spatial/temporal deduplication (optional)
existing = query_captures_within(
    latitude=lat,
    longitude=lon,
    radius_meters=50,
    time_window_hours=1
)

# Upload markers (incremental sync)
last_uploaded_id = get_user_marker(user_id, session_id)
new_captures = filter_captures_after(last_uploaded_id)

3. GPS Handling

Wigle's Quality Checks:

Minimum accuracy threshold: 32 meters
Null Island detection: (0.0, 0.0) rejected
Multi-provider fallback: GPS → Network → Last known
Location interpolation: Fill gaps between observations
Kalman filtering: Smooth noisy GPS tracks

GigLez GPS Validation (from wigle_analysis.md):

def validate_gps(lat: float, lon: float, accuracy: float) -> bool:
    # Bounds check
    if not (-90 <= lat <= 90 and -180 <= lon <= 180):
        return False

    # Null Island check
    if abs(lat) < 0.001 and abs(lon) < 0.001:
        return False

    # Accuracy threshold
    if accuracy > 50:  # meters
        return False

    return True

4. Upload System

Wigle CSV Format:

WigleWifi-1.6,appRelease=2.70,model=Pixel,release=13,device=blueline,...
MAC,SSID,AuthMode,FirstSeen,Channel,Frequency,RSSI,CurrentLatitude,CurrentLongitude,...
00:11:22:33:44:55,MyNetwork,WPA2,2026-01-12 10:00:00,6,2437,-65,40.7128,-74.0060,...

GigLez JSON + Binary Format:

{
  "version": "GigLez-1.0",
  "session_id": "550e8400-e29b-41d4-a716-446655440000",
  "device_info": {
    "model": "LilyGo T-Embed",
    "firmware": "Bruce-2.1",
    "app_version": "1.0.0"
  },
  "captures": [
    {
      "filename": "capture_001.sub",
      "sha256": "abc123...",
      "latitude": 40.7128,
      "longitude": -74.0060,
      "accuracy": 5.0,
      "altitude": 10.5,
      "timestamp": "2026-01-12T10:00:00Z"
    }
  ]
}

5. Performance Optimizations

Wigle's Proven Patterns:

Transaction batching (512 operations per commit)
Prepared statements (avoid SQL parsing overhead)
LRU caching (256 entries for device lookups)
Background threading (offload DB writes)
Pragma optimizations (temp_store=MEMORY, journal_mode=PERSIST)

GigLez Adaptations:

# Transaction batching (PostgreSQL)
with db.begin():
    for i in range(0, len(captures), 512):
        batch = captures[i:i+512]
        db.bulk_insert_mappings(Capture, batch)

# LRU caching (device signatures)
from functools import lru_cache

@lru_cache(maxsize=256)
def get_device_signature(device_id: int):
    return db.query(Device).filter_by(id=device_id).first()

# Async background tasks (FastAPI)
from fastapi import BackgroundTasks

@app.post("/api/captures")
async def upload_captures(files: List[UploadFile], background: BackgroundTasks):
    background.add_task(process_captures, files)
    return {"status": "processing"}

Implementation Roadmap

Phase 1: Foundation (Weeks 1-2)

1.1 Database Setup

Priority: CRITICAL Effort: 2 days

# Install PostgreSQL with PostGIS
sudo apt install postgresql postgresql-contrib postgis

# Create database and user
sudo -u postgres createdb giglez
sudo -u postgres createuser giglez_user -P

# Enable PostGIS extension
psql -d giglez -c "CREATE EXTENSION postgis;"

Tasks:

Install PostgreSQL 14+ and PostGIS 3.x
Create database and user
Convert docs/database_schema.md to SQL script
Add PostGIS geometry columns (ST_SetSRID)
Create spatial indexes (GIST)
Test basic geospatial queries

Deliverable: scripts/create_schema.sql

1.2 SQLAlchemy ORM Models

Priority: CRITICAL Effort: 3 days

# src/database/models.py
from sqlalchemy import Column, String, Integer, Float, DateTime, Text
from sqlalchemy.ext.declarative import declarative_base
from geoalchemy2 import Geometry

Base = declarative_base()

class Capture(Base):
    __tablename__ = 'captures'

    file_hash = Column(String(64), primary_key=True)
    latitude = Column(Float, nullable=False)
    longitude = Column(Float, nullable=False)
    geom = Column(Geometry('POINT', srid=4326))
    timestamp = Column(DateTime, nullable=False)
    frequency = Column(Integer, nullable=False)
    protocol = Column(String(100))
    session_id = Column(String(64))
    file_path = Column(String(500))

Tasks:

Create ORM models for all tables
Add relationship definitions
Implement geom column auto-population (before_insert)
Add validation constraints
Create database migration with Alembic
Write unit tests for model operations

Deliverable: src/database/models.py, alembic/versions/001_initial.py

1.3 GPS Validation Module

Priority: HIGH Effort: 1 day

Copy from docs/wigle_analysis.md Section 10 (GPS validator example).

Tasks:

Create src/gps/validator.py
Implement bounds checking
Implement Null Island detection
Implement accuracy filtering
Add unit tests (valid/invalid cases)

Deliverable: src/gps/validator.py

Phase 2: Upload System (Weeks 3-4)

2.1 FastAPI Application Setup

Priority: CRITICAL Effort: 2 days

# src/api/main.py
from fastapi import FastAPI, UploadFile, File, Form
from fastapi.middleware.cors import CORSMiddleware

app = FastAPI(title="GigLez API", version="1.0.0")

app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],
    allow_methods=["*"],
    allow_headers=["*"]
)

@app.post("/api/captures/upload")
async def upload_captures(
    manifest: str = Form(...),
    files: List[UploadFile] = File(...)
):
    # Parse JSON manifest
    data = json.loads(manifest)

    # Process files
    results = await process_uploads(data, files)

    return {"uploaded": len(results), "captures": results}

Tasks:

Create FastAPI application structure
Add CORS middleware
Implement authentication (JWT + API keys)
Add rate limiting (slowapi)
Configure logging (loguru)
Generate OpenAPI docs

Deliverable: src/api/main.py, src/api/auth.py

2.2 File Upload Endpoint

Priority: CRITICAL Effort: 3 days

Tasks:

Accept multipart/form-data (JSON manifest + .sub files)
Validate manifest schema (JSON schema)
Compute SHA256 hash per file
Check for duplicates before processing
Parse .sub files with existing parser
Validate GPS coordinates
Store files in organized directory structure
Insert captures into database (with transaction batching)
Return upload summary

Deliverable: src/api/routes/captures.py

2.3 Upload Marker System

Priority: MEDIUM Effort: 2 days

From docs/wigle_analysis.md Section 10:

# Track last uploaded capture ID per user/session
class UploadMarker(Base):
    __tablename__ = 'upload_markers'

    user_id = Column(Integer, primary_key=True)
    session_id = Column(String(64), primary_key=True)
    last_capture_id = Column(String(64), nullable=False)
    last_upload_time = Column(DateTime, nullable=False)

# Query for incremental sync
def get_unuploaded_captures(user_id, session_id):
    marker = get_marker(user_id, session_id)
    return query_captures_after(marker.last_capture_id)

Tasks:

Create upload_markers table
Implement marker update on successful upload
Add incremental sync endpoint
Handle resume after failed upload

Deliverable: src/database/markers.py

Phase 3: Device Matching (Weeks 5-6)

3.1 Signature Database Import

Priority: HIGH Effort: 4 days

Tasks:

Clone Flipper Zero firmware repository
Extract .sub files from assets
Parse Flipper signatures (protocol, frequency, bit patterns)
Import into devices and signatures tables
Clone RTL_433 repository
Parse protocol definitions (JSON)
Map RTL_433 fields to database schema
Import test data samples
Create import scripts (idempotent)

Deliverable: scripts/import_flipper.py, scripts/import_rtl433.py

3.2 Matching Engine Implementation

Priority: HIGH Effort: 5 days

From src/matcher/engine.py (already scaffolded), implement strategies:

# src/matcher/strategies.py
class ExactMatchStrategy(MatchStrategy):
    """Exact match: protocol + frequency + bit_length"""
    def match(self, metadata, db):
        results = db.query(
            protocol=metadata.protocol,
            frequency=metadata.frequency,
            bit_length=metadata.bit_length
        )
        return [MatchResult(..., confidence=1.0) for r in results]

class PartialMatchStrategy(MatchStrategy):
    """Partial match: protocol + frequency"""
    def match(self, metadata, db):
        results = db.query(
            protocol=metadata.protocol,
            frequency=metadata.frequency
        )
        return [MatchResult(..., confidence=0.8) for r in results]

class BitPatternStrategy(MatchStrategy):
    """Bit pattern matching with masks"""
    def match(self, metadata, db):
        # Compare key_data against signature bit_patterns
        # Use bit_mask to ignore variable bits
        pass

class TimingStrategy(MatchStrategy):
    """RAW timing pattern matching"""
    def match(self, metadata, db):
        # Extract timing patterns
        # Compare against known timing signatures
        pass

Tasks:

Implement ExactMatchStrategy
Implement PartialMatchStrategy
Implement BitPatternStrategy
Implement TimingStrategy
Add confidence scoring algorithm
Store match results in capture_matches table
Add unit tests for each strategy

Deliverable: src/matcher/strategies.py (complete)

Phase 4: Web Interface (Weeks 7-8)

4.1 Map Visualization

Priority: MEDIUM Effort: 5 days

<!-- src/web/templates/map.html -->
<!DOCTYPE html>
<html>
<head>
    <link rel="stylesheet" href="https://unpkg.com/leaflet@1.9.4/dist/leaflet.css" />
    <script src="https://unpkg.com/leaflet@1.9.4/dist/leaflet.js"></script>
</head>
<body>
    <div id="map" style="height: 600px;"></div>
    <script>
        const map = L.map('map').setView([40.7128, -74.0060], 13);

        L.tileLayer('https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png').addTo(map);

        // Fetch captures and add markers
        fetch('/api/captures?bbox=' + map.getBounds().toBBoxString())
            .then(r => r.json())
            .then(data => {
                data.captures.forEach(capture => {
                    L.marker([capture.latitude, capture.longitude])
                        .bindPopup(`
                            <b>${capture.protocol || 'Unknown'}</b><br>
                            Frequency: ${capture.frequency} Hz<br>
                            Device: ${capture.device_name || 'Unidentified'}
                        `)
                        .addTo(map);
                });
            });
    </script>
</body>
</html>

Tasks:

Set up FastAPI static file serving
Create Leaflet.js map interface
Implement marker clustering (Leaflet.markercluster)
Add heatmap layer (Leaflet.heat)
Create device detail popup
Add search/filter UI
Implement bounding box query optimization

Deliverable: src/web/static/, src/web/templates/

4.2 API Query Endpoints

Priority: HIGH Effort: 3 days

@app.get("/api/captures")
async def query_captures(
    lat: float = None,
    lon: float = None,
    radius_km: float = 1.0,
    bbox: str = None,  # "min_lon,min_lat,max_lon,max_lat"
    protocol: str = None,
    device_id: int = None,
    start_time: datetime = None,
    end_time: datetime = None,
    limit: int = 100
):
    # Build geospatial query
    query = db.query(Capture)

    if lat and lon:
        # Radius query
        query = query.filter(
            func.ST_DWithin(
                Capture.geom,
                func.ST_SetSRID(func.ST_MakePoint(lon, lat), 4326),
                radius_km * 1000
            )
        )
    elif bbox:
        # Bounding box query
        min_lon, min_lat, max_lon, max_lat = map(float, bbox.split(','))
        query = query.filter(
            Capture.geom.ST_Within(
                func.ST_MakeEnvelope(min_lon, min_lat, max_lon, max_lat, 4326)
            )
        )

    # Apply filters
    if protocol:
        query = query.filter(Capture.protocol == protocol)

    results = query.limit(limit).all()
    return {"captures": [c.to_dict() for c in results]}

Tasks:

Implement geospatial queries (radius, bounding box)
Add filtering (protocol, frequency, device, time range)
Implement pagination
Add statistics endpoint (/api/stats)
Add heatmap data endpoint (/api/heatmap)
Add device catalog endpoint (/api/devices)
Optimize queries with indexes

Deliverable: src/api/routes/query.py

Phase 5: Performance & Optimization (Weeks 9-10)

5.1 Transaction Batching

Priority: HIGH Effort: 2 days

From docs/wigle_analysis.md Section 6:

# Batch insert 512 captures per transaction
BATCH_SIZE = 512

def bulk_insert_captures(captures: List[dict]):
    with db.begin():
        for i in range(0, len(captures), BATCH_SIZE):
            batch = captures[i:i+BATCH_SIZE]
            db.bulk_insert_mappings(Capture, batch)

Tasks:

Implement batch insert for captures
Add batch update for existing records
Configure SQLAlchemy connection pooling
Add transaction retry logic
Benchmark performance (before/after)

Deliverable: src/database/batch.py

5.2 Caching Layer

Priority: MEDIUM Effort: 3 days

from functools import lru_cache
import redis

# In-memory LRU cache for device lookups
@lru_cache(maxsize=256)
def get_device(device_id: int):
    return db.query(Device).get(device_id)

# Redis cache for API responses
redis_client = redis.Redis(host='localhost', port=6379, db=0)

@app.get("/api/devices/{device_id}")
async def get_device_api(device_id: int):
    cache_key = f"device:{device_id}"
    cached = redis_client.get(cache_key)

    if cached:
        return json.loads(cached)

    device = db.query(Device).get(device_id)
    redis_client.setex(cache_key, 3600, json.dumps(device.to_dict()))

    return device.to_dict()

Tasks:

Add LRU caching for device/signature lookups
Install Redis for API response caching
Implement cache invalidation strategy
Add cache warming on startup
Monitor cache hit rates

Deliverable: src/cache/, Redis configuration

5.3 Materialized Views

Priority: LOW Effort: 2 days

From docs/database_schema.md (lines 362-401):

-- Pre-compute device statistics
CREATE MATERIALIZED VIEW device_statistics AS
SELECT
    d.id,
    COUNT(c.id) as total_captures,
    MIN(c.timestamp) as first_seen,
    MAX(c.timestamp) as last_seen
FROM devices d
LEFT JOIN captures c ON c.device_id = d.id
GROUP BY d.id;

-- Refresh strategy (cron job)
REFRESH MATERIALIZED VIEW CONCURRENTLY device_statistics;

Tasks:

Create materialized views (device_statistics, geographic_heatmap)
Set up refresh schedule (cron or pg_cron)
Add concurrent refresh to avoid locking
Update API to query views instead of raw tables

Deliverable: scripts/refresh_views.sh, cron configuration

Immediate Next Steps (This Week)

Monday-Tuesday: Database Setup

Install PostgreSQL and PostGIS
Create scripts/create_schema.sql from docs/database_schema.md
Add PostGIS geometry columns
Create spatial indexes
Test basic geospatial queries

Wednesday-Thursday: SQLAlchemy Models

Create src/database/models.py
Implement all table models
Add geom auto-population
Set up Alembic migrations
Write unit tests

Friday: GPS Validation

Create src/gps/validator.py
Implement validation functions from docs/wigle_analysis.md
Add unit tests
Integrate with upload pipeline

Success Metrics

Phase 1 Complete

✅ PostgreSQL database created with PostGIS
✅ All tables created from schema
✅ SQLAlchemy models working
✅ GPS validation module tested
✅ Can insert/query captures programmatically

MVP Complete (Phase 3)

✅ 1000+ signatures imported (Flipper + RTL_433)
✅ Automatic device matching works
✅ API accepts .sub file uploads
✅ Web interface shows captures on map
✅ End-to-end workflow: upload → parse → match → visualize

Production Ready (Phase 5)

✅ Transaction batching implemented (512 ops/commit)
✅ LRU caching for device lookups
✅ Redis caching for API responses
✅ Query performance < 100ms (p95)
✅ Can handle 10,000+ captures/day

Resources & References

Documentation

docs/wigle_analysis.md - Complete Wigle client analysis
docs/architecture_decisions.md - 10 key architectural decisions
docs/database_schema.md - PostgreSQL schema with PostGIS
CLAUDE.md - Project directives and requirements
PROJECT_STATUS.md - Current status and roadmap

Code Examples

GPS validator: docs/wigle_analysis.md Section 10
Upload markers: docs/wigle_analysis.md Section 4
Batch processing: docs/wigle_analysis.md Section 6
LRU caching: docs/architecture_decisions.md Decision 9

External Resources

Wigle Android client: https://github.com/wiglenet/wigle-wifi-wardriving
Wigle API docs: https://api.wigle.net/
PostGIS documentation: https://postgis.net/docs/
FastAPI documentation: https://fastapi.tiangolo.com/
Leaflet.js documentation: https://leafletjs.com/

Risk Mitigation

Technical Risks

Database size growth: Implement archival strategy, compress old captures
Query performance: Add indexes, use materialized views, implement caching
File storage limits: Use object storage (S3/MinIO) for .sub files
GPS accuracy: Follow Wigle's validation patterns (32m threshold)

Implementation Risks

Scope creep: Focus on MVP first (upload → parse → match → visualize)
Over-engineering: Start simple, optimize when needed (measure first)
Signature database maintenance: Automate imports with scripts

Next Action: Start Phase 1 database setup (PostgreSQL + PostGIS installation)

20 KiB Raw Blame History

GigLez Implementation Plan - Post Wigle Analysis

Executive Summary

Current State Assessment

✅ Completed

🚧 In Progress

❌ Not Started

Wigle Analysis Key Takeaways

1. Database Architecture

2. Deduplication Strategy

3. GPS Handling

4. Upload System

5. Performance Optimizations

Implementation Roadmap

Phase 1: Foundation (Weeks 1-2)

1.1 Database Setup

1.2 SQLAlchemy ORM Models

1.3 GPS Validation Module

Phase 2: Upload System (Weeks 3-4)

2.1 FastAPI Application Setup

2.2 File Upload Endpoint

2.3 Upload Marker System

Phase 3: Device Matching (Weeks 5-6)

3.1 Signature Database Import

3.2 Matching Engine Implementation

Phase 4: Web Interface (Weeks 7-8)

4.1 Map Visualization

4.2 API Query Endpoints

Phase 5: Performance & Optimization (Weeks 9-10)

5.1 Transaction Batching

5.2 Caching Layer

5.3 Materialized Views

Immediate Next Steps (This Week)

Monday-Tuesday: Database Setup

Wednesday-Thursday: SQLAlchemy Models

Friday: GPS Validation

Success Metrics

Phase 1 Complete

MVP Complete (Phase 3)

Production Ready (Phase 5)

Resources & References

Documentation

Code Examples

External Resources

Risk Mitigation

Technical Risks

Implementation Risks

20 KiB

Raw Blame History