diff --git a/CONTEXT.md b/CONTEXT.md new file mode 100644 index 0000000..0cf7c53 --- /dev/null +++ b/CONTEXT.md @@ -0,0 +1,34 @@ +# giglez — Project Context + +## Overview +- **Category:** security (RF/SIGINT — Sub-GHz IoT device mapping; moved from media/ 2026-07-07) +- **Status:** stale +- **Tech Stack:** Node.js Python Bash +- **Files:** ~360 files (depth ≤3) +- **Has Tests:** yes +- **Has CI/CD:** no +- **Has CLAUDE.md:** yes +- **Containerized:** no + +## Git +- **Is Git Repo:** yes +- **Branch:** p1-p2-validation +- **Remote:** https://github.com/leetcrypt/giglez.git +- **Last Commit:** docs: add comprehensive CONTRIBUTING.md for GhostArmyIntel collaboration +- **Last Commit Date:** %Y->- (HEAD -> p1-p2-validation, origin/p1-p2-validation) +- **Uncommitted Changes:** no + +## Description +IoT RF Device Mapping Platform - Wigle for Sub-GHz Signals + +## Dependencies +**Requirements:** pyserial,aioserial,psycopg2-binary,sqlalchemy,geoalchemy2,alembic,python-for-android,fastapi, + +## Scripts + - `test`: `python -m pytest tests/ -v` + - `test:decoder`: `node static/js/decoder/test.js` + - `test:python`: `python -m pytest tests/ -v` + - `test:coverage`: `python -m pytest tests/ --cov=src --cov-report=html` + +## Key Files +3_rf_spectrum_analyzer.js,CLAUDE.md,config,CONTRIBUTING.md,create-gitea-repo.sh,data,docs,giglez.db,import_5k_final.log,import_5k_output.log,import_5k_SUCCESS.log,logs,package.json,__pycache__,QUICK_START.md,REAL_TEST_RESULTS.md,requirements.txt,scripts,signatures,src,start_web.sh,static,templates,test_enhanced_matcher.py,test_files diff --git a/FABLE.md b/FABLE.md new file mode 100644 index 0000000..1251a65 --- /dev/null +++ b/FABLE.md @@ -0,0 +1,318 @@ +# GigLez — FABLE Development Brief +**Project**: Sub-GHz IoT RF device mapping platform (Wigle for IoT) +**Repo**: `/home/dell/coding/security/giglez` +**Branch**: `p1-p2-validation` +**Briefed**: 2026-07-17 + +--- + +## Your Job + +You are picking up active development on GigLez. Phase 0 is complete. Your job is to drive Phases 1–2 (API completion + web UI) to a state where a user can open a browser, upload a `.sub` file, and see it appear on a map. That is the MVP gate. + +Read this brief fully before writing a single line of code. The project has a lot of prior work — most of what you'd think to build already exists in some form. Explore before you scaffold. + +--- + +## Situational Awareness + +### What Exists and Works + +| Layer | Files | Confidence | +|-------|-------|-----------| +| `.sub` / RAW parser | `src/parser/sub_parser.py` (308 LOC) | Production-ready | +| GPS validator | `src/gps/validator.py` (22/22 tests pass) | Production-ready | +| Device matcher | `src/matcher/engine.py` → `device_identifier.py` | 67% top-3 real-world | +| Category router | `src/matcher/category_router.py` | Complete (Phase 0) | +| Protocol DB | `src/matcher/protocol_database.py` | 350+ protocols | +| DB schema | `scripts/create_schema.sql` (670 LOC, PostGIS) | Ready to init | +| ORM models | `src/database/models.py` (11 tables) | Complete | +| Storage backends | `src/core/storage/` (FS + S3) | Complete | +| Config system | `config/settings.py` | Dev/prod aware | + +### What's Skeleton Only (Needs You) + +| Layer | Files | Gap | +|-------|-------|-----| +| Upload endpoint | `src/api/routes/captures.py` | 60 LOC of boilerplate, never completed | +| Search endpoint | `src/api/routes/query.py` | 4-line stub | +| Stats endpoint | `src/api/routes/stats.py` | 3-line stub | +| Devices endpoint | `src/api/routes/devices.py` | 5-line stub | +| FastAPI app | `src/api/main_simple.py` | Uses MockDB, old strategies | +| Web UI | `templates/`, `static/js/` | Empty structures | +| Auth | `src/api/dependencies.py` | `optional_auth` stub exists | + +### Duplicate/Dead Files — Ignore These + +- `src/api/main_orm_legacy.py` — superseded +- `src/api/routes/captures_enhanced.py` — superseded by `captures.py` +- `src/api/routes/captures_with_logging.py` — superseded +- `src/matcher/strategies.py` — legacy fallback, don't extend + +--- + +## Architecture Snapshot + +``` +Browser / CLI client + │ + ▼ +FastAPI (src/api/main_simple.py → currently MockDB mode) + │ + ├── POST /api/v1/captures/upload + │ → parse_sub_file() → validate_gps() → matcher.match() → db.store() + │ + ├── GET /api/v1/search?bbox=...&freq=...&category=... + │ → PostGIS ST_Within → returns GeoJSON + │ + ├── GET /api/v1/heatmap + │ → materialized view capture_heatmap → grid densities + │ + ├── GET /api/v1/stats + │ → materialized view device_statistics + │ + └── GET /api/v1/devices + → known device catalog + +Database: PostgreSQL + PostGIS + Schema at: scripts/create_schema.sql + Dev: SQLite fallback acceptable for MVP + Prod: PostgreSQL required + +Storage: src/core/storage/factory.py + Dev: STORAGE_BACKEND=filesystem (local ./data/uploads/) + Prod: STORAGE_BACKEND=s3 +``` + +--- + +## Phase 1: Complete the Backend API + +**Goal**: `POST /api/v1/captures/upload` → stored in DB → returns match results. + +### 1.1 Reconcile the FastAPI entry point + +`src/api/main_simple.py` currently uses `MockDB` and imports the old strategy classes. You need to: +- Replace `MockDB` with the real DB session from `config/database.py` +- Wire in the proper router pattern +- Make the app runnable in "no-DB" dev mode (SQLite or in-memory) so you don't block on PostgreSQL setup +- Keep the existing `/docs` and `/redoc` endpoints + +**Dev mode fallback**: If `DATABASE_URL` is not set, use `sqlite:///./giglez.db` (already exists at project root from prior testing). The ORM models already use SQLAlchemy 2.0, so this is a connection string swap. + +### 1.2 Complete `POST /api/v1/captures/upload` + +The skeleton is at `src/api/routes/captures.py:29`. It has the right shape — manifest parsing, file hashing — but never stores or matches. Complete it: + +``` +receive multipart (manifest JSON + .sub files) + → validate manifest schema (pydantic) + → for each file: + sha256 = hash(bytes) + check duplicate (Capture.file_hash in db) → skip if exists + metadata = parse_sub_file(tmp_path) + gps = validate_gps_coordinates(lat, lon, accuracy) + matches = SignatureMatcher().match(metadata, max_results=5) + file_url = storage.save(sha256, bytes) + db.add(Capture(...)) + db.add_all([CaptureMatch(...) for top 3 matches]) + → return { + processed: int, + skipped_duplicates: int, + captures: [{id, lat, lon, matches: [{device, confidence, category}]}] + } +``` + +Use `SignatureMatcher` from `src/matcher/engine.py`, not the old strategies. + +### 1.3 Complete `GET /api/v1/search` + +```python +@router.get("/search") +async def search_captures( + lat: float, lon: float, radius_km: float = 1.0, + frequency_min: Optional[int] = None, + frequency_max: Optional[int] = None, + category: Optional[str] = None, + limit: int = 100, + db: Session = Depends(get_db), +): + # PostGIS: ST_DWithin(geometry, ST_SetSRID(ST_MakePoint(lon, lat), 4326), radius_m) + # Return GeoJSON FeatureCollection for Leaflet +``` + +For the SQLite dev fallback, do a simple lat/lon bounding box (haversine already in `src/gps/validator.py`). + +### 1.4 Complete `GET /api/v1/stats` and `GET /api/v1/devices` + +Stats: total captures, unique device types, geographic coverage (distinct countries/cities), top 5 protocols. +Devices: paginated list of known device types with capture counts. + +### 1.5 Auth middleware + +`src/api/dependencies.py` has `optional_auth` stub. For MVP: +- Anonymous uploads: allowed, rate-limited by IP (use `slowapi` or simple in-memory counter) +- API key (optional): `X-API-Key` header, stored in `users.api_key` column +- No JWT yet — defer to Phase 5 + +--- + +## Phase 2: Web UI MVP + +**Goal**: User opens browser → sees map → drags `.sub` file → sees it appear as a marker. + +Approach: Jinja2 templates + Leaflet.js via CDN. No build step. No framework. Vanilla JS. + +### 2.1 Templates needed + +``` +templates/ +├── base.html ← nav, CDN links (Leaflet 1.9, Tailwind CDN, htmx 1.9) +├── index.html ← landing: stats summary + recent 10 captures on mini-map +├── map.html ← full-screen Leaflet map + filter sidebar +└── upload.html ← drag-and-drop upload form +``` + +Use **htmx** for the upload form — it makes multipart upload + response handling trivial without writing JS. The upload endpoint returns partial HTML with match results that htmx swaps in. + +### 2.2 Leaflet map (map.html) + +```javascript +const map = L.map('map').setView([39.5, -98.35], 4); // USA center +L.tileLayer('https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png').addTo(map); + +// Marker cluster +const markers = L.markerClusterGroup(); + +// Load from API +fetch('/api/v1/search?lat=39.5&lon=-98.35&radius_km=5000') + .then(r => r.json()) + .then(geojson => { + L.geoJSON(geojson, { + onEachFeature: (feature, layer) => { + layer.bindPopup(` + ${feature.properties.device_name}
+ ${feature.properties.frequency_mhz} MHz
+ ${feature.properties.category}
+ Confidence: ${feature.properties.confidence}% + `); + } + }).addTo(markers); + map.addLayer(markers); + }); +``` + +CDN links needed: +- `https://unpkg.com/leaflet@1.9.4/dist/leaflet.css` +- `https://unpkg.com/leaflet@1.9.4/dist/leaflet.js` +- `https://unpkg.com/leaflet.markercluster@1.5.3/dist/MarkerCluster.css` +- `https://unpkg.com/leaflet.markercluster@1.5.3/dist/leaflet.markercluster.js` +- `https://cdn.tailwindcss.com` +- `https://unpkg.com/htmx.org@1.9.10` + +### 2.3 Upload form (upload.html) + +```html +
+ + + + + + +
+
+``` + +The upload endpoint should return an HTML partial (not JSON) when `Accept: text/html` — show matches as cards. + +--- + +## Key Technical Constraints + +1. **Don't touch `src/matcher/`** except to import from it. The Phase 0 accuracy work is complete. The only valid change is importing the new `SignatureMatcher` in the API. + +2. **Don't add Redis yet**. Rate limiting can be in-memory dict for MVP. Redis is Phase 4. + +3. **No JWT yet**. API key + anonymous is sufficient for MVP. + +4. **SQLite dev mode is mandatory**. Not everyone has PostgreSQL running. The app must start with `uvicorn src.api.main:app --reload` and work without any external services. + +5. **Don't create `requirements.txt` additions unless strictly necessary**. Everything needed is already installed: fastapi, uvicorn, sqlalchemy, jinja2, pydantic, loguru, numpy, scipy, scikit-learn, geopy. + +6. **`htmx`** for the upload form only — don't add it to the map page. Leaflet is sufficient there. + +--- + +## How to Run / Test + +```bash +# From project root +cd /home/dell/coding/security/giglez + +# Dev server (SQLite, no DB setup needed) +DATABASE_URL=sqlite:///./giglez.db uvicorn src.api.main:app --reload + +# Run existing tests +python3 -m pytest tests/unit/ -q + +# Phase 0 accuracy benchmark +python3 scripts/benchmark_phase0.py + +# Quick matcher smoke test +python3 -c " +from src.matcher.engine import SignatureMatcher +from src.parser.sub_parser import parse_sub_file +# ... test with a .sub file from signatures/t-embed-rf/ +" +``` + +--- + +## How to Use Your Capabilities Here + +You are FABLE — Opus 4.6, the most capable model in the Claude lineup. Use that accordingly: + +**Parallelize ruthlessly.** Phase 1 API routes and Phase 2 templates are independent. Use the Agent tool to run them simultaneously: one subagent completing all 4 API routes while another writes the 4 templates. You should be orchestrating, not writing every line sequentially yourself. + +**Make architectural calls, don't ask.** If you see a better pattern than what's described here — use it. The constraints above are floors, not ceilings. If you see a cleaner way to structure the upload response, ship it. + +**Read before writing.** Before touching any file, read it. The project has multiple versions of things (captures.py, captures_enhanced.py, captures_with_logging.py). The right one is `captures.py`. Read the 60 existing lines before adding to them. + +**Fix the right thing.** The upload endpoint is the #1 priority. If you complete nothing else, complete `POST /api/v1/captures/upload` end-to-end with a working SQLite fallback. A user being able to upload and get match results back is the first tangible milestone. + +**The map is the moat.** The moment captures appear as markers on a Leaflet map is when GigLez becomes a real product rather than a Python library. Prioritize that moment. + +--- + +## Success Criteria + +| Milestone | Criteria | +|-----------|----------| +| Phase 1 done | `curl -X POST .../captures/upload -F manifest=... -F files=@test.sub` returns match JSON | +| Phase 2 done | `http://localhost:8000/map` shows markers for uploaded captures | +| MVP gate | Both above, running on SQLite, no PostgreSQL required | + +--- + +## Related Files to Read First + +``` +PLAN_TO_PROD.md ← Full production roadmap +REAL_TEST_RESULTS.md ← Baseline accuracy data (before Phase 0) +src/api/main_simple.py ← Current FastAPI entry point +src/api/routes/captures.py ← Upload endpoint skeleton +src/api/dependencies.py ← Auth + DB dependency stubs +src/database/models.py ← All ORM models +config/settings.py ← Environment-aware settings +config/database.py ← DB connection management +scripts/create_schema.sql ← PostgreSQL schema (reference) +.env.development ← Dev environment template +``` + +--- + +*Last updated: 2026-07-17 after Phase 0 completion (0% → 67% top-3 real-world accuracy).* diff --git a/PLAN_TO_PROD.md b/PLAN_TO_PROD.md new file mode 100644 index 0000000..aa06d76 --- /dev/null +++ b/PLAN_TO_PROD.md @@ -0,0 +1,512 @@ +# GigLez — Plan to Production +**Analyzed**: 2026-07-17 +**Branch**: p1-p2-validation +**Status**: Backend 65% complete · Frontend 5% · Real-world accuracy 0% + +--- + +## Critical Findings (Read This First) + +### The 0% Real-World Problem + +The most important finding from benchmark analysis: + +| Benchmark | Top-1 Accuracy | Source | +|-----------|---------------|--------| +| Synthetic signals | 33.3% | `scripts/benchmark.py` | +| **Real Flipper Zero captures** | **0.0%** | `REAL_TEST_RESULTS.md` | + +The matcher produces high-confidence wrong answers on real signals (e.g., LaCrosse TX141 → "Schou 72543 Rain Gauge" at 69% confidence). This is **not a threshold issue — it is a fundamental database gap**. The RTL_433 protocol set (260+ entries) is dominated by weather sensor timing signatures, so everything pattern-matches to a weather sensor regardless of what it actually is. + +**Root cause**: The `rtl433_protocols_imported.py` (4540 LOC, the matching backbone) contains only open/known protocols broadcast continuously. Real-world targets like garage door openers, fan controllers, and LED remotes use **rolling codes (KeeLoq), proprietary encodings, or undocumented bit structures** that have no signatures to match against. These account for 60%+ of Sub-GHz traffic. + +Fixing this is the first gating requirement for everything else. + +--- + +## Project Snapshot + +### What's Built (Do Not Rebuild) + +| Component | File(s) | Status | Quality | +|-----------|---------|--------|---------| +| `.sub` / RAW file parser | `src/parser/sub_parser.py` | ✅ Complete | Production-ready | +| GPS validator | `src/gps/validator.py` | ✅ Complete | 22/22 tests pass | +| ORM models (11 tables) | `src/database/models.py` | ✅ Complete | PostGIS ready | +| PostgreSQL schema | `scripts/create_schema.sql` | ✅ Complete | Full triggers/indexes | +| Storage abstraction | `src/core/storage/` | ✅ Complete | FS + S3 backends | +| Matching engine scaffolding | `src/matcher/engine.py` | ✅ Structure | Needs accuracy fix | +| 5-method scorer | `src/matcher/device_identifier.py` | ✅ Built | Wrong outputs | +| RTL_433 protocol DB | `src/matcher/rtl433_protocols_imported.py` | ✅ Imported | Incomplete coverage | +| T-Embed hardware controller | `src/capture/tembed.py` | ✅ Built | Untested live | +| FastAPI app skeleton | `src/api/main_simple.py` | 🚧 Partial | Routes incomplete | +| Configuration system | `config/settings.py` | ✅ Complete | Dev/prod aware | +| ML design doc | `docs/ML_DESIGN.md` | ✅ Designed | Not implemented | + +### What's Missing + +| Component | Priority | Effort | +|-----------|----------|--------| +| Accuracy fix (device category DB + protocol routing) | P0 | 2 days | +| ML hybrid ensemble (CNN + heuristic) | P1 | 1 week | +| Complete API endpoints | P1 | 3 days | +| Web UI (map + upload form) | P1 | 1 week | +| User auth (JWT + API keys) | P2 | 2 days | +| Docker + deployment | P2 | 1 day | +| Community features (voting, leaderboard) | P3 | 1 week | +| CI/CD pipeline | P3 | 1 day | + +--- + +## Device Identification: What Works, What Doesn't + +### Identifiable Today (with fixes applied) + +These device classes have open, documented protocols with stable timing signatures that the current architecture can match: + +| Device Class | Protocols | Accuracy Potential | Frequency | +|-------------|-----------|-------------------|-----------| +| Weather stations | Oregon Scientific v1/v2/v3, Acurite, LaCrosse | 70-85% | 433.92 MHz | +| TPMS (tire pressure) | Schrader, Continental, Pacific, Citroen | 60-75% | 315/433 MHz | +| Simple door/window sensors | EV1527, PT2262/2272, Princeton | 65-80% | 315/433 MHz | +| Temperature/humidity sensors | Nexus, Ambient Weather, Bresser | 70-80% | 433 MHz | +| Rain gauges | Multiple RTL_433-supported | 65-75% | 433 MHz | +| Thermostats (simple) | Honeywell, Braeburn | 55-70% | 345/433 MHz | +| Smart meters (EU) | Kamstrup, Landis+Gyr | 60-70% | 868 MHz | +| LoRa devices (type only) | LoRaWAN chirp signature | 50-60% | 868/915 MHz | + +**Total identifiable universe**: ~350-400 known device types via RTL_433 + Flipper DB + +### Cannot Be Identified (Protocol-Level Limitation) + +| Device Class | Why | Workaround | +|-------------|-----|-----------| +| Rolling code garage doors | KeeLoq / Security+ 2.0 — every transmission unique | Can identify device class/family, not specific code | +| Modern car key fobs | Hopping codes | Frequency + timing only → brand inference | +| Z-Wave devices | Proprietary 9-byte network ID | Preamble detection → "Z-Wave" category | +| Zigbee (2.4 GHz) | Out of Sub-GHz range | See 2.4 GHz section below | +| Fan/LED remotes | Undocumented proprietary | ML category classification helps | +| Somfy RTS | Proprietary encrypted | Frequency match only (433.42 MHz = Somfy) | +| DECT cordless phones | 1.88-1.9 GHz | Out of range | + +### Identification Approach by Protocol Type + +``` +Signal received + │ + ├─ Has Protocol field (decoded .sub) → Heuristic match → 80-95% accuracy + │ + ├─ RAW data, known timing signature → RTL_433 protocol match → 60-80% + │ + ├─ RAW data, unknown timing → ML CNN classifier → 50-70% (device category) + │ + ├─ Frequency-only match → ISM band category → 30-50% (class only) + │ + └─ No match → "Unknown Sub-GHz @ {freq}" → stored for community ID +``` + +--- + +## Machine Learning Integration + +### Why ML Is Needed + +The heuristic matcher covers open protocols well. ML fills the gap for: +1. **RAW signals** with no protocol match (fan controllers, LED remotes, proprietary) +2. **Device category classification** when exact ID is impossible (rolling codes) +3. **Manufacturer inference** from RF hardware fingerprints +4. **Novel protocol discovery** — clustering unknown signals + +### Recommended Architecture (from `docs/ML_DESIGN.md`) + +Implement the designed **Hybrid Ensemble** in phases: + +#### Phase A: Statistical Features (Lowest effort, highest ROI) + +Train a **Random Forest / Gradient Boosting classifier** on 47 extracted features: +- Pulse width statistics (mean, std, min, max SHORT/LONG) +- LONG/SHORT ratio +- Pulse count, repetition count +- Preamble pattern type +- ISM band classification +- Duty cycle, inter-burst gap + +```python +# Feature vector already partially built in: +# src/matcher/statistical_classifier.py (356 LOC) +# src/matcher/timing_analyzer.py (550 LOC) + +# Target: device category (not model) +# Classes: weather_sensor, garage_door, key_fob, doorbell, +# security_sensor, fan_controller, tpms, smart_meter, unknown +``` + +**Training data**: UberGuidoZ Flipper DB (10,000+ labeled `.sub` files) — directory structure provides labels. +**Expected accuracy**: 60-75% category-level +**Effort**: 3-4 days (feature extraction exists, need training pipeline) + +#### Phase B: 1D CNN on Raw Pulse Arrays + +``` +Input: normalized pulse array → padded to 512 samples +Architecture: + Conv1D(64, 3) → BatchNorm → ReLU + Conv1D(128, 3) → BatchNorm → ReLU + Conv1D(256, 3) → BatchNorm → ReLU + GlobalAvgPool + Dense(256) → Dropout(0.3) + Dense(N_classes) → Softmax + +Output: P(device_category) for 9 categories +``` + +**Training**: PyTorch or TensorFlow (Python training, ONNX export for serving) +**Inference**: Python via ONNX Runtime (server-side, NOT browser — model too complex for TF.js) +**Expected accuracy**: 70-80% category-level +**Effort**: 5-7 days + +#### Phase C: Ensemble Combiner + +```python +final_score = ( + 0.40 * heuristic_score + # existing matcher (exact protocols) + 0.35 * cnn_score + # CNN category probability + 0.25 * stat_score # statistical features +) +``` + +The heuristic matcher dominates for known protocols; ML fills gaps for unknowns. + +#### What ML Cannot Do + +- Decode rolling code content (cryptographically protected) +- Identify specific device model from RAW alone (insufficient features) +- Work on BinRAW without normalization +- Achieve >90% model-level accuracy with Sub-GHz RAW (physics limitation) + +**Realistic production targets**: +- Known protocol devices: 80-90% top-1 accuracy (heuristic) +- Unknown/proprietary: 60-75% device category (ML) +- Rolling code: 40-55% manufacturer/family inference + +--- + +## 2.4 GHz / 5 GHz / 6 GHz Feasibility + +### Technical Reality + +| Band | Hardware Needed | Protocol | File Format | Flipper Support | +|------|----------------|----------|-------------|----------------| +| Sub-GHz (300-928 MHz) | Flipper Zero, RTL-SDR | OOK/FSK/ASK | `.sub` | ✅ Native | +| 2.4 GHz | WiFi card (monitor mode) | 802.11 b/g/n, Zigbee, BT | `.pcap` / `.pcapng` | ❌ Hardware limit | +| 5 GHz | WiFi card (monitor mode) | 802.11 a/n/ac | `.pcap` | ❌ Hardware limit | +| 6 GHz | WiFi 6E card | 802.11ax | `.pcap` | ❌ Hardware limit | + +### Can GigLez Support 2.4/5/6 GHz? + +**Yes, but as a separate ingestion path.** The platform architecture (GPS + file + database) generalizes cleanly. What changes: + +1. **Input format**: Accept `.pcap`/`.pcapng` instead of `.sub` +2. **Parser**: `pyshark` or `scapy` to extract beacon/probe frames (already in `~/coding/_archive/pyshark`) +3. **Identification**: Network fingerprinting by: + - SSID patterns → device type (e.g., `DECO-XXXX` → TP-Link mesh) + - OUI lookup from MAC → manufacturer + - IE (Information Elements) → chipset inference +4. **Database**: Separate `wifi_captures` table alongside `captures` + +**Wigle does exactly this for WiFi** — their schema can be adapted directly. + +**Recommendation**: Phase 4 feature. Sub-GHz is the differentiator; WiFi is Wigle's territory. Add 2.4/5/6 GHz support after MVP to expand data collection surface, not before. + +**Effort**: 1 week to add WiFi pcap ingestion + OUI database + map layer. + +--- + +## Production Roadmap + +### Phase 0: Fix Accuracy (Gating — 1 week) + +This blocks everything. No point shipping a product that gives wrong answers with high confidence. + +**0.1 — Build device category routing layer** (2 days) + +Create `src/matcher/category_router.py` that classifies *before* trying to match: +```python +# Frequency + timing → category → route to specialized sub-matcher +# 315 MHz + short TE < 500 → key_fob_matcher +# 433 MHz + long burst preamble → weather_matcher +# 433 MHz + EV1527 timing → remote_matcher +``` + +**0.2 — Expand signature database** (2 days) + +The current DB has RTL_433 protocols. Add: +- Princeton, EV1527, PT2262 parameter ranges (simple remotes — largest category) +- KeeLoq family detection (rolling code → don't match model, classify family) +- Flipper Zero firmware `.sub` collection bulk import (UberGuidoZ: 10K+ files) + - Parse filenames as labels: `Garage_Doors/LiftMaster/*.sub` → category + brand +- Z-Wave preamble signature (868/908 MHz chirp pattern) + +**0.3 — Add confidence calibration** (1 day) + +Current system produces 69-76% confidence for wrong answers. Add isotonic regression calibration or simply apply hard rules: +- If top-1 and top-2 confidence differ by < 5% → reduce both to "low" +- If frequency doesn't match device category → penalize 30% +- If timing LONG/SHORT ratio is outside protocol's known range → cap at 50% + +**0.4 — Re-run real-world benchmark** (0.5 days) + +Target: Top-3 accuracy ≥ 30% on `REAL_TEST_RESULTS.md` test set. This is achievable with routing alone. + +--- + +### Phase 1: Complete the Backend API (1 week) + +All routes exist as skeletons. Wire them up. + +**1.1 — `POST /api/v1/captures/upload`** (2 days) +- Accept multipart: JSON manifest + `.sub` files +- Validate GPS (already done — `src/gps/validator.py`) +- Parse files (already done — `src/parser/sub_parser.py`) +- Run matcher (fixed in Phase 0) +- Store in PostgreSQL +- Return capture IDs + match results + +**1.2 — `GET /api/v1/search`** (1 day) +- Query by bounding box (PostGIS `ST_Within`) +- Filter by: frequency range, protocol, device category, date range +- Pagination (cursor-based, Wigle pattern) +- Return GeoJSON for map rendering + +**1.3 — `GET /api/v1/heatmap`** (0.5 days) +- Aggregate by grid cell (H3 hexagons or lat/lon bins) +- Use materialized view `capture_heatmap` (already designed in schema) +- Return density + dominant device category per cell + +**1.4 — `GET /api/v1/stats`** (0.5 days) +- Total captures, unique devices, geographic coverage +- Top frequencies, protocols, device categories +- Uses materialized view `device_statistics` + +**1.5 — `GET /api/v1/devices`** (1 day) +- List/search known device catalog +- Filter by manufacturer, category, frequency +- Device detail page (signatures, captures seen) + +**1.6 — Auth middleware** (1 day) +- Anonymous uploads allowed (rate-limited by IP) +- Optional JWT for account-linked uploads +- API key header for programmatic access +- FastAPI dependency injection (clean pattern) + +--- + +### Phase 2: Web Interface MVP (1 week) + +Minimum: something users can open in a browser and actually use. + +**2.1 — Upload form** (2 days) +- Drag-and-drop `.sub` files +- GPS entry: manual lat/lon OR "use browser location" +- Timestamp auto-fill +- Upload progress + results display +- Simple HTML/CSS + vanilla JS (no framework needed for MVP) + +**2.2 — Interactive map** (3 days) +- Leaflet.js (already in plan, free, no API key) +- Marker clustering (`Leaflet.markercluster`) +- Click → device details panel +- Heatmap toggle (`Leaflet.heat`) +- Filter sidebar: frequency, category, date + +**2.3 — Basic pages** (2 days) +- `/` — Landing + recent captures +- `/map` — Interactive map +- `/upload` — Upload form +- `/devices` — Device catalog +- `/stats` — Platform statistics + +Use Jinja2 templates (FastAPI native) + Tailwind CSS via CDN. No build step needed. + +--- + +### Phase 3: ML Integration (1 week, parallel-able with Phase 2) + +Implement Phase A + B from the ML strategy above. + +**3.1 — Training data pipeline** (2 days) +```bash +# Download UberGuidoZ Flipper database (public, ~2GB) +# Parse directory structure as labels +# Extract 47 statistical features per file +# Split 80/10/10 train/val/test +``` + +**3.2 — Train statistical classifier** (1 day) +```python +# sklearn GradientBoostingClassifier or XGBoost +# 9 device categories +# Save as joblib model +# Expected: 60-75% category accuracy +``` + +**3.3 — Train 1D CNN** (2 days) +```python +# PyTorch, pad RAW arrays to 512 samples +# Train 20 epochs on GPU (if available) or CPU (~2h) +# Export to ONNX +# Expected: 70-80% category accuracy +``` + +**3.4 — Wire ensemble into matcher** (1 day) +- Load ONNX model at startup +- Run in parallel with heuristic matcher (async) +- Combine scores with 40/35/25 weighting +- Store ML category alongside heuristic matches in `capture_matches` table + +**3.5 — Active learning hook** (0.5 days) +- Community verifications → retrain queue +- Simple: flag captures with low confidence + community correction +- Retrain monthly + +--- + +### Phase 4: Dockerize + Deploy (3 days) + +**4.1 — Docker Compose** (1 day) + +```yaml +services: + api: # FastAPI + uvicorn + db: # PostgreSQL 16 + PostGIS + redis: # Cache + rate limiting + worker: # Celery for async file processing + nginx: # Reverse proxy + static files +``` + +**4.2 — Deployment target** (1 day) +- VPS: 4 vCPU, 8GB RAM, 100GB SSD (enough for 10M captures) +- PostgreSQL tuned for geospatial workloads +- Nginx + Let's Encrypt SSL +- systemd service for auto-restart + +**4.3 — Operational tooling** (1 day) +- Health check endpoint +- Prometheus metrics (`/metrics`) +- Log rotation (loguru already wired) +- DB backup cron (daily dump to S3) + +--- + +### Phase 5: Community Features (2 weeks, post-launch) + +**5.1 — User accounts** (3 days) +- Registration/login (email or anonymous) +- Upload history + statistics +- API key management +- Reputation score + +**5.2 — Device verification** (3 days) +- "Confirm" / "Dispute" buttons on matches +- Voting system (votes table already in schema) +- High-vote matches → promoted to "verified" +- Community corrections feed ML retraining + +**5.3 — Leaderboards + badges** (2 days) +- Top uploaders (by count, by new devices discovered) +- Geographic coverage leader +- Milestone badges (first in city, 100 captures, etc.) + +**5.4 — 2.4/5 GHz expansion** (5 days) +- Accept `.pcap`/`.pcapng` uploads +- `pyshark`-based parser for WiFi beacons +- OUI database lookup (manufacturer from MAC) +- New map layer: WiFi devices +- Wigle-compatible CSV export + +--- + +## Tech Stack Gap Analysis + +### Has / Doesn't Need Replacing + +| Component | Current | Status | +|-----------|---------|--------| +| Parser | Custom Python | ✅ Keep | +| GPS validation | Custom Python | ✅ Keep | +| Database | PostgreSQL + PostGIS | ✅ Keep (needs init) | +| ORM | SQLAlchemy 2.0 | ✅ Keep | +| API framework | FastAPI 0.109 | ✅ Keep | +| Storage | FS (dev) / S3 (prod) | ✅ Keep | +| Config | python-dotenv | ✅ Keep | + +### Needs Adding + +| Component | Recommendation | Why | +|-----------|---------------|-----| +| Map library | Leaflet.js 1.9 | Free, no API key, proven | +| ML training | PyTorch + scikit-learn | Already in requirements | +| ML inference | ONNX Runtime | Lightweight, no GPU required | +| Caching | Redis (via `redis-py`) | Rate limiting + query cache | +| Task queue | Celery + Redis | Async file processing | +| Frontend CSS | Tailwind CSS (CDN) | No build step for MVP | +| Container | Docker + Compose | Reproducible deployment | +| Reverse proxy | Nginx | SSL + static files | + +### Remove / Simplify + +| Item | Action | +|------|--------| +| `src/api/main_orm_legacy.py` | Delete — superseded | +| `src/api/routes/captures_enhanced.py` | Merge into `captures.py`, delete | +| `src/api/routes/captures_with_logging.py` | Merge into `captures.py`, delete | +| `src/matcher/strategies.py` (610 LOC legacy) | Keep as fallback but don't extend | +| `scripts/archive/` | Leave archived — don't delete | + +--- + +## Honest Timeline to Production + +Assuming 4-6 focused hours/day: + +| Phase | What | Duration | Cumulative | +|-------|------|----------|------------| +| 0 | Fix accuracy (critical) | 1 week | Week 1 | +| 1 | Complete backend API | 1 week | Week 2 | +| 2 | Web UI MVP | 1 week | Week 3 | +| 3 | ML integration | 1 week | Week 3-4 (parallel) | +| 4 | Docker + deploy | 3 days | Week 4 | +| **MVP Live** | **Public beta** | | **~Week 4** | +| 5 | Community features | 2 weeks | Week 6 | +| WiFi | 2.4/5 GHz expansion | 1 week | Week 7 | + +**Minimum viable launch** (something to show GhostArmyIntel): **3-4 weeks** + +--- + +## Key External Resources for Phase 0 + +| Resource | URL | Use | +|----------|-----|-----| +| UberGuidoZ Flipper DB | `github.com/UberGuidoZ/Flipper` | 10K+ labeled .sub training files | +| RTL_433 test files | `github.com/merbanan/rtl_433/tree/master/tests` | Labeled real captures | +| RadioML dataset | `deepsig.ai/datasets` | ML training for modulation recognition | +| Protocol table | `github.com/merbanan/rtl_433/blob/master/README.md` | 260+ protocol reference | +| Flipper firmware protocol list | `github.com/flipperdevices/flipperzero-firmware/tree/dev/lib/subghz/protocols` | Canonical protocol params | + +--- + +## Success Criteria (Per Phase) + +| Phase | Gate | +|-------|------| +| Phase 0 complete | Real-world top-3 accuracy ≥ 30% (up from 0%) | +| Phase 1 complete | `POST /upload` → stored in DB → returns matches | +| Phase 2 complete | User can upload .sub file in browser, see it on map | +| Phase 3 complete | ML model reduces "unknown" rate from 45% → < 25% | +| Phase 4 complete | `docker compose up` → production site live on VPS | +| Phase 5 complete | 100+ community-verified identifications | + +--- + +*Generated from deep analysis of project state + `docs/ML_DESIGN.md` + `REAL_TEST_RESULTS.md` + `docs/RF_SIGNAL_ANALYSIS_RESEARCH.md` + `docs/FREQUENCY_DEVICE_CHART.md` on 2026-07-17.* diff --git a/src/api/main_simple.py b/src/api/main_simple.py index 37c6375..1ccc1d2 100644 --- a/src/api/main_simple.py +++ b/src/api/main_simple.py @@ -395,6 +395,13 @@ async def upload_captures( sub_parser = SubFileParser() gps_extractor = GPSFilenameExtractor() + # Map manifest entries by filename so each file gets its own GPS/timestamp + # (previously every file was assigned captures[0], breaking multi-file uploads) + manifest_map = { + c.get("filename"): c + for c in manifest_data.get("captures", []) + } + successful = [] failed = [] @@ -429,6 +436,18 @@ async def upload_captures( # Get best match from unified results best_match = match_results[0] if match_results else None + # Per-file manifest entry (falls back to first entry, then empty) + entry = manifest_map.get(file.filename) + if entry is None: + entry = manifest_data.get("captures", [{}])[0] + + # The real device category comes from the category router (e.g. + # "Weather Sensor", "Remote Control"), carried in match_details. + best_category = ( + best_match.match_details.get("predicted_category") + if best_match else None + ) + # Use GPS from manifest or filename global upload_counter upload_counter += 1 @@ -439,15 +458,15 @@ async def upload_captures( "frequency": frequency, "protocol": protocol, "preset": preset, - "latitude": gps_coords.latitude if gps_coords else manifest_data.get("captures", [{}])[0].get("latitude"), - "longitude": gps_coords.longitude if gps_coords else manifest_data.get("captures", [{}])[0].get("longitude"), + "latitude": gps_coords.latitude if gps_coords else entry.get("latitude"), + "longitude": gps_coords.longitude if gps_coords else entry.get("longitude"), "gps_source": gps_coords.source if gps_coords else "manual", - "timestamp": manifest_data.get("captures", [{}])[0].get("timestamp", ""), + "timestamp": entry.get("timestamp", ""), "data_source": manifest_data.get("data_source", "production"), # production, test, or mock "session_id": manifest_data.get("session_uuid", ""), # Device identification fields (from unified matcher) "device_name": best_match.device_name if best_match else None, - "device_category": f"{best_match.manufacturer} - {best_match.device_name}" if best_match else None, + "device_category": best_category, "match_confidence": best_match.confidence if best_match else None, "match_method": best_match.match_method if best_match else None, "device_description": f"{best_match.manufacturer} {best_match.device_name}" if best_match else None, @@ -456,6 +475,7 @@ async def upload_captures( { "device_name": m.device_name, "manufacturer": m.manufacturer, + "category": m.match_details.get("predicted_category"), "confidence": m.confidence, "method": m.match_method, "details": m.match_details