fix: correct per-file GPS and device category in upload handler

- main_simple.py upload: map manifest entries by filename so each file gets its own GPS/timestamp (previously every file was assigned captures[0], breaking multi-file uploads) - device_category now uses the category router's real category (e.g. "Remote Control") instead of the mislabeled "manufacturer - device_name" string; also surface category per matched device - add PLAN_TO_PROD.md / FABLE.md development briefs - add CONTEXT.md (leaked PAT redacted from remote URL) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-18 13:09:52 -07:00
parent 1b9b33fa3f
commit 78c28fe43f
4 changed files with 888 additions and 4 deletions
@@ -0,0 +1,34 @@
+# giglez — Project Context
+
+## Overview
+- **Category:** security (RF/SIGINT — Sub-GHz IoT device mapping; moved from media/ 2026-07-07)
+- **Status:** stale
+- **Tech Stack:** Node.js Python Bash
+- **Files:** ~360 files (depth ≤3)
+- **Has Tests:** yes
+- **Has CI/CD:** no
+- **Has CLAUDE.md:** yes
+- **Containerized:** no
+
+## Git
+- **Is Git Repo:** yes
+- **Branch:** p1-p2-validation
+- **Remote:** https://github.com/leetcrypt/giglez.git
+- **Last Commit:** docs: add comprehensive CONTRIBUTING.md for GhostArmyIntel collaboration
+- **Last Commit Date:** %Y->- (HEAD -> p1-p2-validation, origin/p1-p2-validation)
+- **Uncommitted Changes:** no
+
+## Description
+IoT RF Device Mapping Platform - Wigle for Sub-GHz Signals
+
+## Dependencies
+**Requirements:** pyserial,aioserial,psycopg2-binary,sqlalchemy,geoalchemy2,alembic,python-for-android,fastapi,
+
+## Scripts
+  - `test`: `python -m pytest tests/ -v`
+  - `test:decoder`: `node static/js/decoder/test.js`
+  - `test:python`: `python -m pytest tests/ -v`
+  - `test:coverage`: `python -m pytest tests/ --cov=src --cov-report=html`
+
+## Key Files
+3_rf_spectrum_analyzer.js,CLAUDE.md,config,CONTRIBUTING.md,create-gitea-repo.sh,data,docs,giglez.db,import_5k_final.log,import_5k_output.log,import_5k_SUCCESS.log,logs,package.json,__pycache__,QUICK_START.md,REAL_TEST_RESULTS.md,requirements.txt,scripts,signatures,src,start_web.sh,static,templates,test_enhanced_matcher.py,test_files
@@ -0,0 +1,318 @@
+# GigLez — FABLE Development Brief
+**Project**: Sub-GHz IoT RF device mapping platform (Wigle for IoT)  
+**Repo**: `/home/dell/coding/security/giglez`  
+**Branch**: `p1-p2-validation`  
+**Briefed**: 2026-07-17
+
+---
+
+## Your Job
+
+You are picking up active development on GigLez. Phase 0 is complete. Your job is to drive Phases 1–2 (API completion + web UI) to a state where a user can open a browser, upload a `.sub` file, and see it appear on a map. That is the MVP gate.
+
+Read this brief fully before writing a single line of code. The project has a lot of prior work — most of what you'd think to build already exists in some form. Explore before you scaffold.
+
+---
+
+## Situational Awareness
+
+### What Exists and Works
+
+| Layer | Files | Confidence |
+|-------|-------|-----------|
+| `.sub` / RAW parser | `src/parser/sub_parser.py` (308 LOC) | Production-ready |
+| GPS validator | `src/gps/validator.py` (22/22 tests pass) | Production-ready |
+| Device matcher | `src/matcher/engine.py` → `device_identifier.py` | 67% top-3 real-world |
+| Category router | `src/matcher/category_router.py` | Complete (Phase 0) |
+| Protocol DB | `src/matcher/protocol_database.py` | 350+ protocols |
+| DB schema | `scripts/create_schema.sql` (670 LOC, PostGIS) | Ready to init |
+| ORM models | `src/database/models.py` (11 tables) | Complete |
+| Storage backends | `src/core/storage/` (FS + S3) | Complete |
+| Config system | `config/settings.py` | Dev/prod aware |
+
+### What's Skeleton Only (Needs You)
+
+| Layer | Files | Gap |
+|-------|-------|-----|
+| Upload endpoint | `src/api/routes/captures.py` | 60 LOC of boilerplate, never completed |
+| Search endpoint | `src/api/routes/query.py` | 4-line stub |
+| Stats endpoint | `src/api/routes/stats.py` | 3-line stub |
+| Devices endpoint | `src/api/routes/devices.py` | 5-line stub |
+| FastAPI app | `src/api/main_simple.py` | Uses MockDB, old strategies |
+| Web UI | `templates/`, `static/js/` | Empty structures |
+| Auth | `src/api/dependencies.py` | `optional_auth` stub exists |
+
+### Duplicate/Dead Files — Ignore These
+
+- `src/api/main_orm_legacy.py` — superseded
+- `src/api/routes/captures_enhanced.py` — superseded by `captures.py`
+- `src/api/routes/captures_with_logging.py` — superseded
+- `src/matcher/strategies.py` — legacy fallback, don't extend
+
+---
+
+## Architecture Snapshot
+
+```
+Browser / CLI client
+        │
+        ▼
+FastAPI (src/api/main_simple.py → currently MockDB mode)
+        │
+        ├── POST /api/v1/captures/upload
+        │       → parse_sub_file() → validate_gps() → matcher.match() → db.store()
+        │
+        ├── GET  /api/v1/search?bbox=...&freq=...&category=...
+        │       → PostGIS ST_Within → returns GeoJSON
+        │
+        ├── GET  /api/v1/heatmap
+        │       → materialized view capture_heatmap → grid densities
+        │
+        ├── GET  /api/v1/stats
+        │       → materialized view device_statistics
+        │
+        └── GET  /api/v1/devices
+                → known device catalog
+
+Database: PostgreSQL + PostGIS
+  Schema at: scripts/create_schema.sql
+  Dev: SQLite fallback acceptable for MVP
+  Prod: PostgreSQL required
+
+Storage: src/core/storage/factory.py
+  Dev:  STORAGE_BACKEND=filesystem (local ./data/uploads/)
+  Prod: STORAGE_BACKEND=s3
+```
+
+---
+
+## Phase 1: Complete the Backend API
+
+**Goal**: `POST /api/v1/captures/upload` → stored in DB → returns match results.
+
+### 1.1 Reconcile the FastAPI entry point
+
+`src/api/main_simple.py` currently uses `MockDB` and imports the old strategy classes. You need to:
+- Replace `MockDB` with the real DB session from `config/database.py`
+- Wire in the proper router pattern
+- Make the app runnable in "no-DB" dev mode (SQLite or in-memory) so you don't block on PostgreSQL setup
+- Keep the existing `/docs` and `/redoc` endpoints
+
+**Dev mode fallback**: If `DATABASE_URL` is not set, use `sqlite:///./giglez.db` (already exists at project root from prior testing). The ORM models already use SQLAlchemy 2.0, so this is a connection string swap.
+
+### 1.2 Complete `POST /api/v1/captures/upload`
+
+The skeleton is at `src/api/routes/captures.py:29`. It has the right shape — manifest parsing, file hashing — but never stores or matches. Complete it:
+
+```
+receive multipart (manifest JSON + .sub files)
+    → validate manifest schema (pydantic)
+    → for each file:
+        sha256 = hash(bytes)
+        check duplicate (Capture.file_hash in db) → skip if exists
+        metadata = parse_sub_file(tmp_path)
+        gps = validate_gps_coordinates(lat, lon, accuracy)
+        matches = SignatureMatcher().match(metadata, max_results=5)
+        file_url = storage.save(sha256, bytes)
+        db.add(Capture(...))
+        db.add_all([CaptureMatch(...) for top 3 matches])
+    → return {
+        processed: int,
+        skipped_duplicates: int,
+        captures: [{id, lat, lon, matches: [{device, confidence, category}]}]
+      }
+```
+
+Use `SignatureMatcher` from `src/matcher/engine.py`, not the old strategies.
+
+### 1.3 Complete `GET /api/v1/search`
+
+```python
+@router.get("/search")
+async def search_captures(
+    lat: float, lon: float, radius_km: float = 1.0,
+    frequency_min: Optional[int] = None,
+    frequency_max: Optional[int] = None,
+    category: Optional[str] = None,
+    limit: int = 100,
+    db: Session = Depends(get_db),
+):
+    # PostGIS: ST_DWithin(geometry, ST_SetSRID(ST_MakePoint(lon, lat), 4326), radius_m)
+    # Return GeoJSON FeatureCollection for Leaflet
+```
+
+For the SQLite dev fallback, do a simple lat/lon bounding box (haversine already in `src/gps/validator.py`).
+
+### 1.4 Complete `GET /api/v1/stats` and `GET /api/v1/devices`
+
+Stats: total captures, unique device types, geographic coverage (distinct countries/cities), top 5 protocols.  
+Devices: paginated list of known device types with capture counts.
+
+### 1.5 Auth middleware
+
+`src/api/dependencies.py` has `optional_auth` stub. For MVP:
+- Anonymous uploads: allowed, rate-limited by IP (use `slowapi` or simple in-memory counter)
+- API key (optional): `X-API-Key` header, stored in `users.api_key` column
+- No JWT yet — defer to Phase 5
+
+---
+
+## Phase 2: Web UI MVP
+
+**Goal**: User opens browser → sees map → drags `.sub` file → sees it appear as a marker.
+
+Approach: Jinja2 templates + Leaflet.js via CDN. No build step. No framework. Vanilla JS.
+
+### 2.1 Templates needed
+
+```
+templates/
+├── base.html       ← nav, CDN links (Leaflet 1.9, Tailwind CDN, htmx 1.9)
+├── index.html      ← landing: stats summary + recent 10 captures on mini-map
+├── map.html        ← full-screen Leaflet map + filter sidebar
+└── upload.html     ← drag-and-drop upload form
+```
+
+Use **htmx** for the upload form — it makes multipart upload + response handling trivial without writing JS. The upload endpoint returns partial HTML with match results that htmx swaps in.
+
+### 2.2 Leaflet map (map.html)
+
+```javascript
+const map = L.map('map').setView([39.5, -98.35], 4);  // USA center
+L.tileLayer('https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png').addTo(map);
+
+// Marker cluster
+const markers = L.markerClusterGroup();
+
+// Load from API
+fetch('/api/v1/search?lat=39.5&lon=-98.35&radius_km=5000')
+  .then(r => r.json())
+  .then(geojson => {
+    L.geoJSON(geojson, {
+      onEachFeature: (feature, layer) => {
+        layer.bindPopup(`
+          <b>${feature.properties.device_name}</b><br>
+          ${feature.properties.frequency_mhz} MHz<br>
+          ${feature.properties.category}<br>
+          Confidence: ${feature.properties.confidence}%
+        `);
+      }
+    }).addTo(markers);
+    map.addLayer(markers);
+  });
+```
+
+CDN links needed:
+- `https://unpkg.com/leaflet@1.9.4/dist/leaflet.css`
+- `https://unpkg.com/leaflet@1.9.4/dist/leaflet.js`
+- `https://unpkg.com/leaflet.markercluster@1.5.3/dist/MarkerCluster.css`
+- `https://unpkg.com/leaflet.markercluster@1.5.3/dist/leaflet.markercluster.js`
+- `https://cdn.tailwindcss.com`
+- `https://unpkg.com/htmx.org@1.9.10`
+
+### 2.3 Upload form (upload.html)
+
+```html
+<form hx-post="/api/v1/captures/upload"
+      hx-encoding="multipart/form-data"
+      hx-target="#results"
+      hx-swap="innerHTML">
+  <input type="hidden" name="manifest" id="manifest-json">
+  <input type="file" name="files" multiple accept=".sub">
+  <!-- GPS inputs with "Use my location" button -->
+  <input name="latitude" placeholder="Latitude">
+  <input name="longitude" placeholder="Longitude">
+  <button>Upload</button>
+</form>
+<div id="results"></div>
+```
+
+The upload endpoint should return an HTML partial (not JSON) when `Accept: text/html` — show matches as cards.
+
+---
+
+## Key Technical Constraints
+
+1. **Don't touch `src/matcher/`** except to import from it. The Phase 0 accuracy work is complete. The only valid change is importing the new `SignatureMatcher` in the API.
+
+2. **Don't add Redis yet**. Rate limiting can be in-memory dict for MVP. Redis is Phase 4.
+
+3. **No JWT yet**. API key + anonymous is sufficient for MVP.
+
+4. **SQLite dev mode is mandatory**. Not everyone has PostgreSQL running. The app must start with `uvicorn src.api.main:app --reload` and work without any external services.
+
+5. **Don't create `requirements.txt` additions unless strictly necessary**. Everything needed is already installed: fastapi, uvicorn, sqlalchemy, jinja2, pydantic, loguru, numpy, scipy, scikit-learn, geopy.
+
+6. **`htmx`** for the upload form only — don't add it to the map page. Leaflet is sufficient there.
+
+---
+
+## How to Run / Test
+
+```bash
+# From project root
+cd /home/dell/coding/security/giglez
+
+# Dev server (SQLite, no DB setup needed)
+DATABASE_URL=sqlite:///./giglez.db uvicorn src.api.main:app --reload
+
+# Run existing tests
+python3 -m pytest tests/unit/ -q
+
+# Phase 0 accuracy benchmark
+python3 scripts/benchmark_phase0.py
+
+# Quick matcher smoke test
+python3 -c "
+from src.matcher.engine import SignatureMatcher
+from src.parser.sub_parser import parse_sub_file
+# ... test with a .sub file from signatures/t-embed-rf/
+"
+```
+
+---
+
+## How to Use Your Capabilities Here
+
+You are FABLE — Opus 4.6, the most capable model in the Claude lineup. Use that accordingly:
+
+**Parallelize ruthlessly.** Phase 1 API routes and Phase 2 templates are independent. Use the Agent tool to run them simultaneously: one subagent completing all 4 API routes while another writes the 4 templates. You should be orchestrating, not writing every line sequentially yourself.
+
+**Make architectural calls, don't ask.** If you see a better pattern than what's described here — use it. The constraints above are floors, not ceilings. If you see a cleaner way to structure the upload response, ship it.
+
+**Read before writing.** Before touching any file, read it. The project has multiple versions of things (captures.py, captures_enhanced.py, captures_with_logging.py). The right one is `captures.py`. Read the 60 existing lines before adding to them.
+
+**Fix the right thing.** The upload endpoint is the #1 priority. If you complete nothing else, complete `POST /api/v1/captures/upload` end-to-end with a working SQLite fallback. A user being able to upload and get match results back is the first tangible milestone.
+
+**The map is the moat.** The moment captures appear as markers on a Leaflet map is when GigLez becomes a real product rather than a Python library. Prioritize that moment.
+
+---
+
+## Success Criteria
+
+| Milestone | Criteria |
+|-----------|----------|
+| Phase 1 done | `curl -X POST .../captures/upload -F manifest=... -F files=@test.sub` returns match JSON |
+| Phase 2 done | `http://localhost:8000/map` shows markers for uploaded captures |
+| MVP gate | Both above, running on SQLite, no PostgreSQL required |
+
+---
+
+## Related Files to Read First
+
+```
+PLAN_TO_PROD.md              ← Full production roadmap
+REAL_TEST_RESULTS.md         ← Baseline accuracy data (before Phase 0)
+src/api/main_simple.py       ← Current FastAPI entry point
+src/api/routes/captures.py   ← Upload endpoint skeleton
+src/api/dependencies.py      ← Auth + DB dependency stubs
+src/database/models.py       ← All ORM models
+config/settings.py           ← Environment-aware settings
+config/database.py           ← DB connection management
+scripts/create_schema.sql    ← PostgreSQL schema (reference)
+.env.development             ← Dev environment template
+```
+
+---
+
+*Last updated: 2026-07-17 after Phase 0 completion (0% → 67% top-3 real-world accuracy).*
@@ -0,0 +1,512 @@
+# GigLez — Plan to Production
+**Analyzed**: 2026-07-17  
+**Branch**: p1-p2-validation  
+**Status**: Backend 65% complete · Frontend 5% · Real-world accuracy 0%
+
+---
+
+## Critical Findings (Read This First)
+
+### The 0% Real-World Problem
+
+The most important finding from benchmark analysis:
+
+| Benchmark | Top-1 Accuracy | Source |
+|-----------|---------------|--------|
+| Synthetic signals | 33.3% | `scripts/benchmark.py` |
+| **Real Flipper Zero captures** | **0.0%** | `REAL_TEST_RESULTS.md` |
+
+The matcher produces high-confidence wrong answers on real signals (e.g., LaCrosse TX141 → "Schou 72543 Rain Gauge" at 69% confidence). This is **not a threshold issue — it is a fundamental database gap**. The RTL_433 protocol set (260+ entries) is dominated by weather sensor timing signatures, so everything pattern-matches to a weather sensor regardless of what it actually is.
+
+**Root cause**: The `rtl433_protocols_imported.py` (4540 LOC, the matching backbone) contains only open/known protocols broadcast continuously. Real-world targets like garage door openers, fan controllers, and LED remotes use **rolling codes (KeeLoq), proprietary encodings, or undocumented bit structures** that have no signatures to match against. These account for 60%+ of Sub-GHz traffic.
+
+Fixing this is the first gating requirement for everything else.
+
+---
+
+## Project Snapshot
+
+### What's Built (Do Not Rebuild)
+
+| Component | File(s) | Status | Quality |
+|-----------|---------|--------|---------|
+| `.sub` / RAW file parser | `src/parser/sub_parser.py` | ✅ Complete | Production-ready |
+| GPS validator | `src/gps/validator.py` | ✅ Complete | 22/22 tests pass |
+| ORM models (11 tables) | `src/database/models.py` | ✅ Complete | PostGIS ready |
+| PostgreSQL schema | `scripts/create_schema.sql` | ✅ Complete | Full triggers/indexes |
+| Storage abstraction | `src/core/storage/` | ✅ Complete | FS + S3 backends |
+| Matching engine scaffolding | `src/matcher/engine.py` | ✅ Structure | Needs accuracy fix |
+| 5-method scorer | `src/matcher/device_identifier.py` | ✅ Built | Wrong outputs |
+| RTL_433 protocol DB | `src/matcher/rtl433_protocols_imported.py` | ✅ Imported | Incomplete coverage |
+| T-Embed hardware controller | `src/capture/tembed.py` | ✅ Built | Untested live |
+| FastAPI app skeleton | `src/api/main_simple.py` | 🚧 Partial | Routes incomplete |
+| Configuration system | `config/settings.py` | ✅ Complete | Dev/prod aware |
+| ML design doc | `docs/ML_DESIGN.md` | ✅ Designed | Not implemented |
+
+### What's Missing
+
+| Component | Priority | Effort |
+|-----------|----------|--------|
+| Accuracy fix (device category DB + protocol routing) | P0 | 2 days |
+| ML hybrid ensemble (CNN + heuristic) | P1 | 1 week |
+| Complete API endpoints | P1 | 3 days |
+| Web UI (map + upload form) | P1 | 1 week |
+| User auth (JWT + API keys) | P2 | 2 days |
+| Docker + deployment | P2 | 1 day |
+| Community features (voting, leaderboard) | P3 | 1 week |
+| CI/CD pipeline | P3 | 1 day |
+
+---
+
+## Device Identification: What Works, What Doesn't
+
+### Identifiable Today (with fixes applied)
+
+These device classes have open, documented protocols with stable timing signatures that the current architecture can match:
+
+| Device Class | Protocols | Accuracy Potential | Frequency |
+|-------------|-----------|-------------------|-----------|
+| Weather stations | Oregon Scientific v1/v2/v3, Acurite, LaCrosse | 70-85% | 433.92 MHz |
+| TPMS (tire pressure) | Schrader, Continental, Pacific, Citroen | 60-75% | 315/433 MHz |
+| Simple door/window sensors | EV1527, PT2262/2272, Princeton | 65-80% | 315/433 MHz |
+| Temperature/humidity sensors | Nexus, Ambient Weather, Bresser | 70-80% | 433 MHz |
+| Rain gauges | Multiple RTL_433-supported | 65-75% | 433 MHz |
+| Thermostats (simple) | Honeywell, Braeburn | 55-70% | 345/433 MHz |
+| Smart meters (EU) | Kamstrup, Landis+Gyr | 60-70% | 868 MHz |
+| LoRa devices (type only) | LoRaWAN chirp signature | 50-60% | 868/915 MHz |
+
+**Total identifiable universe**: ~350-400 known device types via RTL_433 + Flipper DB
+
+### Cannot Be Identified (Protocol-Level Limitation)
+
+| Device Class | Why | Workaround |
+|-------------|-----|-----------|
+| Rolling code garage doors | KeeLoq / Security+ 2.0 — every transmission unique | Can identify device class/family, not specific code |
+| Modern car key fobs | Hopping codes | Frequency + timing only → brand inference |
+| Z-Wave devices | Proprietary 9-byte network ID | Preamble detection → "Z-Wave" category |
+| Zigbee (2.4 GHz) | Out of Sub-GHz range | See 2.4 GHz section below |
+| Fan/LED remotes | Undocumented proprietary | ML category classification helps |
+| Somfy RTS | Proprietary encrypted | Frequency match only (433.42 MHz = Somfy) |
+| DECT cordless phones | 1.88-1.9 GHz | Out of range |
+
+### Identification Approach by Protocol Type
+
+```
+Signal received
+    │
+    ├─ Has Protocol field (decoded .sub) → Heuristic match → 80-95% accuracy
+    │
+    ├─ RAW data, known timing signature → RTL_433 protocol match → 60-80%
+    │
+    ├─ RAW data, unknown timing → ML CNN classifier → 50-70% (device category)
+    │
+    ├─ Frequency-only match → ISM band category → 30-50% (class only)
+    │
+    └─ No match → "Unknown Sub-GHz @ {freq}" → stored for community ID
+```
+
+---
+
+## Machine Learning Integration
+
+### Why ML Is Needed
+
+The heuristic matcher covers open protocols well. ML fills the gap for:
+1. **RAW signals** with no protocol match (fan controllers, LED remotes, proprietary)
+2. **Device category classification** when exact ID is impossible (rolling codes)  
+3. **Manufacturer inference** from RF hardware fingerprints
+4. **Novel protocol discovery** — clustering unknown signals
+
+### Recommended Architecture (from `docs/ML_DESIGN.md`)
+
+Implement the designed **Hybrid Ensemble** in phases:
+
+#### Phase A: Statistical Features (Lowest effort, highest ROI)
+
+Train a **Random Forest / Gradient Boosting classifier** on 47 extracted features:
+- Pulse width statistics (mean, std, min, max SHORT/LONG)
+- LONG/SHORT ratio
+- Pulse count, repetition count
+- Preamble pattern type
+- ISM band classification
+- Duty cycle, inter-burst gap
+
+```python
+# Feature vector already partially built in:
+# src/matcher/statistical_classifier.py (356 LOC)
+# src/matcher/timing_analyzer.py (550 LOC)
+
+# Target: device category (not model)
+# Classes: weather_sensor, garage_door, key_fob, doorbell, 
+#          security_sensor, fan_controller, tpms, smart_meter, unknown
+```
+
+**Training data**: UberGuidoZ Flipper DB (10,000+ labeled `.sub` files) — directory structure provides labels.  
+**Expected accuracy**: 60-75% category-level  
+**Effort**: 3-4 days (feature extraction exists, need training pipeline)
+
+#### Phase B: 1D CNN on Raw Pulse Arrays
+
+```
+Input: normalized pulse array → padded to 512 samples
+Architecture:
+  Conv1D(64, 3) → BatchNorm → ReLU
+  Conv1D(128, 3) → BatchNorm → ReLU  
+  Conv1D(256, 3) → BatchNorm → ReLU
+  GlobalAvgPool
+  Dense(256) → Dropout(0.3)
+  Dense(N_classes) → Softmax
+
+Output: P(device_category) for 9 categories
+```
+
+**Training**: PyTorch or TensorFlow (Python training, ONNX export for serving)  
+**Inference**: Python via ONNX Runtime (server-side, NOT browser — model too complex for TF.js)  
+**Expected accuracy**: 70-80% category-level  
+**Effort**: 5-7 days
+
+#### Phase C: Ensemble Combiner
+
+```python
+final_score = (
+    0.40 * heuristic_score +   # existing matcher (exact protocols)
+    0.35 * cnn_score +          # CNN category probability  
+    0.25 * stat_score           # statistical features
+)
+```
+
+The heuristic matcher dominates for known protocols; ML fills gaps for unknowns.
+
+#### What ML Cannot Do
+
+- Decode rolling code content (cryptographically protected)
+- Identify specific device model from RAW alone (insufficient features)
+- Work on BinRAW without normalization
+- Achieve >90% model-level accuracy with Sub-GHz RAW (physics limitation)
+
+**Realistic production targets**:
+- Known protocol devices: 80-90% top-1 accuracy (heuristic)
+- Unknown/proprietary: 60-75% device category (ML)
+- Rolling code: 40-55% manufacturer/family inference
+
+---
+
+## 2.4 GHz / 5 GHz / 6 GHz Feasibility
+
+### Technical Reality
+
+| Band | Hardware Needed | Protocol | File Format | Flipper Support |
+|------|----------------|----------|-------------|----------------|
+| Sub-GHz (300-928 MHz) | Flipper Zero, RTL-SDR | OOK/FSK/ASK | `.sub` | ✅ Native |
+| 2.4 GHz | WiFi card (monitor mode) | 802.11 b/g/n, Zigbee, BT | `.pcap` / `.pcapng` | ❌ Hardware limit |
+| 5 GHz | WiFi card (monitor mode) | 802.11 a/n/ac | `.pcap` | ❌ Hardware limit |
+| 6 GHz | WiFi 6E card | 802.11ax | `.pcap` | ❌ Hardware limit |
+
+### Can GigLez Support 2.4/5/6 GHz?
+
+**Yes, but as a separate ingestion path.** The platform architecture (GPS + file + database) generalizes cleanly. What changes:
+
+1. **Input format**: Accept `.pcap`/`.pcapng` instead of `.sub`
+2. **Parser**: `pyshark` or `scapy` to extract beacon/probe frames (already in `~/coding/_archive/pyshark`)
+3. **Identification**: Network fingerprinting by:
+   - SSID patterns → device type (e.g., `DECO-XXXX` → TP-Link mesh)
+   - OUI lookup from MAC → manufacturer
+   - IE (Information Elements) → chipset inference
+4. **Database**: Separate `wifi_captures` table alongside `captures`
+
+**Wigle does exactly this for WiFi** — their schema can be adapted directly.
+
+**Recommendation**: Phase 4 feature. Sub-GHz is the differentiator; WiFi is Wigle's territory. Add 2.4/5/6 GHz support after MVP to expand data collection surface, not before.
+
+**Effort**: 1 week to add WiFi pcap ingestion + OUI database + map layer.
+
+---
+
+## Production Roadmap
+
+### Phase 0: Fix Accuracy (Gating — 1 week)
+
+This blocks everything. No point shipping a product that gives wrong answers with high confidence.
+
+**0.1 — Build device category routing layer** (2 days)
+
+Create `src/matcher/category_router.py` that classifies *before* trying to match:
+```python
+# Frequency + timing → category → route to specialized sub-matcher
+# 315 MHz + short TE < 500 → key_fob_matcher
+# 433 MHz + long burst preamble → weather_matcher  
+# 433 MHz + EV1527 timing → remote_matcher
+```
+
+**0.2 — Expand signature database** (2 days)
+
+The current DB has RTL_433 protocols. Add:
+- Princeton, EV1527, PT2262 parameter ranges (simple remotes — largest category)
+- KeeLoq family detection (rolling code → don't match model, classify family)
+- Flipper Zero firmware `.sub` collection bulk import (UberGuidoZ: 10K+ files)
+  - Parse filenames as labels: `Garage_Doors/LiftMaster/*.sub` → category + brand
+- Z-Wave preamble signature (868/908 MHz chirp pattern)
+
+**0.3 — Add confidence calibration** (1 day)
+
+Current system produces 69-76% confidence for wrong answers. Add isotonic regression calibration or simply apply hard rules:
+- If top-1 and top-2 confidence differ by < 5% → reduce both to "low"  
+- If frequency doesn't match device category → penalize 30%
+- If timing LONG/SHORT ratio is outside protocol's known range → cap at 50%
+
+**0.4 — Re-run real-world benchmark** (0.5 days)
+
+Target: Top-3 accuracy ≥ 30% on `REAL_TEST_RESULTS.md` test set. This is achievable with routing alone.
+
+---
+
+### Phase 1: Complete the Backend API (1 week)
+
+All routes exist as skeletons. Wire them up.
+
+**1.1 — `POST /api/v1/captures/upload`** (2 days)
+- Accept multipart: JSON manifest + `.sub` files
+- Validate GPS (already done — `src/gps/validator.py`)
+- Parse files (already done — `src/parser/sub_parser.py`)
+- Run matcher (fixed in Phase 0)
+- Store in PostgreSQL
+- Return capture IDs + match results
+
+**1.2 — `GET /api/v1/search`** (1 day)
+- Query by bounding box (PostGIS `ST_Within`)
+- Filter by: frequency range, protocol, device category, date range
+- Pagination (cursor-based, Wigle pattern)
+- Return GeoJSON for map rendering
+
+**1.3 — `GET /api/v1/heatmap`** (0.5 days)
+- Aggregate by grid cell (H3 hexagons or lat/lon bins)
+- Use materialized view `capture_heatmap` (already designed in schema)
+- Return density + dominant device category per cell
+
+**1.4 — `GET /api/v1/stats`** (0.5 days)
+- Total captures, unique devices, geographic coverage
+- Top frequencies, protocols, device categories
+- Uses materialized view `device_statistics`
+
+**1.5 — `GET /api/v1/devices`** (1 day)
+- List/search known device catalog
+- Filter by manufacturer, category, frequency
+- Device detail page (signatures, captures seen)
+
+**1.6 — Auth middleware** (1 day)
+- Anonymous uploads allowed (rate-limited by IP)
+- Optional JWT for account-linked uploads
+- API key header for programmatic access
+- FastAPI dependency injection (clean pattern)
+
+---
+
+### Phase 2: Web Interface MVP (1 week)
+
+Minimum: something users can open in a browser and actually use.
+
+**2.1 — Upload form** (2 days)
+- Drag-and-drop `.sub` files
+- GPS entry: manual lat/lon OR "use browser location"
+- Timestamp auto-fill
+- Upload progress + results display
+- Simple HTML/CSS + vanilla JS (no framework needed for MVP)
+
+**2.2 — Interactive map** (3 days)
+- Leaflet.js (already in plan, free, no API key)
+- Marker clustering (`Leaflet.markercluster`)
+- Click → device details panel
+- Heatmap toggle (`Leaflet.heat`)
+- Filter sidebar: frequency, category, date
+
+**2.3 — Basic pages** (2 days)
+- `/` — Landing + recent captures
+- `/map` — Interactive map
+- `/upload` — Upload form
+- `/devices` — Device catalog
+- `/stats` — Platform statistics
+
+Use Jinja2 templates (FastAPI native) + Tailwind CSS via CDN. No build step needed.
+
+---
+
+### Phase 3: ML Integration (1 week, parallel-able with Phase 2)
+
+Implement Phase A + B from the ML strategy above.
+
+**3.1 — Training data pipeline** (2 days)
+```bash
+# Download UberGuidoZ Flipper database (public, ~2GB)
+# Parse directory structure as labels
+# Extract 47 statistical features per file
+# Split 80/10/10 train/val/test
+```
+
+**3.2 — Train statistical classifier** (1 day)
+```python
+# sklearn GradientBoostingClassifier or XGBoost
+# 9 device categories
+# Save as joblib model
+# Expected: 60-75% category accuracy
+```
+
+**3.3 — Train 1D CNN** (2 days)
+```python
+# PyTorch, pad RAW arrays to 512 samples  
+# Train 20 epochs on GPU (if available) or CPU (~2h)
+# Export to ONNX
+# Expected: 70-80% category accuracy
+```
+
+**3.4 — Wire ensemble into matcher** (1 day)
+- Load ONNX model at startup
+- Run in parallel with heuristic matcher (async)
+- Combine scores with 40/35/25 weighting
+- Store ML category alongside heuristic matches in `capture_matches` table
+
+**3.5 — Active learning hook** (0.5 days)
+- Community verifications → retrain queue
+- Simple: flag captures with low confidence + community correction
+- Retrain monthly
+
+---
+
+### Phase 4: Dockerize + Deploy (3 days)
+
+**4.1 — Docker Compose** (1 day)
+
+```yaml
+services:
+  api:        # FastAPI + uvicorn
+  db:         # PostgreSQL 16 + PostGIS
+  redis:      # Cache + rate limiting
+  worker:     # Celery for async file processing
+  nginx:      # Reverse proxy + static files
+```
+
+**4.2 — Deployment target** (1 day)
+- VPS: 4 vCPU, 8GB RAM, 100GB SSD (enough for 10M captures)
+- PostgreSQL tuned for geospatial workloads
+- Nginx + Let's Encrypt SSL
+- systemd service for auto-restart
+
+**4.3 — Operational tooling** (1 day)
+- Health check endpoint
+- Prometheus metrics (`/metrics`)
+- Log rotation (loguru already wired)
+- DB backup cron (daily dump to S3)
+
+---
+
+### Phase 5: Community Features (2 weeks, post-launch)
+
+**5.1 — User accounts** (3 days)
+- Registration/login (email or anonymous)
+- Upload history + statistics
+- API key management
+- Reputation score
+
+**5.2 — Device verification** (3 days)
+- "Confirm" / "Dispute" buttons on matches
+- Voting system (votes table already in schema)
+- High-vote matches → promoted to "verified"
+- Community corrections feed ML retraining
+
+**5.3 — Leaderboards + badges** (2 days)
+- Top uploaders (by count, by new devices discovered)
+- Geographic coverage leader
+- Milestone badges (first in city, 100 captures, etc.)
+
+**5.4 — 2.4/5 GHz expansion** (5 days)
+- Accept `.pcap`/`.pcapng` uploads
+- `pyshark`-based parser for WiFi beacons
+- OUI database lookup (manufacturer from MAC)
+- New map layer: WiFi devices
+- Wigle-compatible CSV export
+
+---
+
+## Tech Stack Gap Analysis
+
+### Has / Doesn't Need Replacing
+
+| Component | Current | Status |
+|-----------|---------|--------|
+| Parser | Custom Python | ✅ Keep |
+| GPS validation | Custom Python | ✅ Keep |
+| Database | PostgreSQL + PostGIS | ✅ Keep (needs init) |
+| ORM | SQLAlchemy 2.0 | ✅ Keep |
+| API framework | FastAPI 0.109 | ✅ Keep |
+| Storage | FS (dev) / S3 (prod) | ✅ Keep |
+| Config | python-dotenv | ✅ Keep |
+
+### Needs Adding
+
+| Component | Recommendation | Why |
+|-----------|---------------|-----|
+| Map library | Leaflet.js 1.9 | Free, no API key, proven |
+| ML training | PyTorch + scikit-learn | Already in requirements |
+| ML inference | ONNX Runtime | Lightweight, no GPU required |
+| Caching | Redis (via `redis-py`) | Rate limiting + query cache |
+| Task queue | Celery + Redis | Async file processing |
+| Frontend CSS | Tailwind CSS (CDN) | No build step for MVP |
+| Container | Docker + Compose | Reproducible deployment |
+| Reverse proxy | Nginx | SSL + static files |
+
+### Remove / Simplify
+
+| Item | Action |
+|------|--------|
+| `src/api/main_orm_legacy.py` | Delete — superseded |
+| `src/api/routes/captures_enhanced.py` | Merge into `captures.py`, delete |
+| `src/api/routes/captures_with_logging.py` | Merge into `captures.py`, delete |
+| `src/matcher/strategies.py` (610 LOC legacy) | Keep as fallback but don't extend |
+| `scripts/archive/` | Leave archived — don't delete |
+
+---
+
+## Honest Timeline to Production
+
+Assuming 4-6 focused hours/day:
+
+| Phase | What | Duration | Cumulative |
+|-------|------|----------|------------|
+| 0 | Fix accuracy (critical) | 1 week | Week 1 |
+| 1 | Complete backend API | 1 week | Week 2 |
+| 2 | Web UI MVP | 1 week | Week 3 |
+| 3 | ML integration | 1 week | Week 3-4 (parallel) |
+| 4 | Docker + deploy | 3 days | Week 4 |
+| **MVP Live** | **Public beta** | | **~Week 4** |
+| 5 | Community features | 2 weeks | Week 6 |
+| WiFi | 2.4/5 GHz expansion | 1 week | Week 7 |
+
+**Minimum viable launch** (something to show GhostArmyIntel): **3-4 weeks**
+
+---
+
+## Key External Resources for Phase 0
+
+| Resource | URL | Use |
+|----------|-----|-----|
+| UberGuidoZ Flipper DB | `github.com/UberGuidoZ/Flipper` | 10K+ labeled .sub training files |
+| RTL_433 test files | `github.com/merbanan/rtl_433/tree/master/tests` | Labeled real captures |
+| RadioML dataset | `deepsig.ai/datasets` | ML training for modulation recognition |
+| Protocol table | `github.com/merbanan/rtl_433/blob/master/README.md` | 260+ protocol reference |
+| Flipper firmware protocol list | `github.com/flipperdevices/flipperzero-firmware/tree/dev/lib/subghz/protocols` | Canonical protocol params |
+
+---
+
+## Success Criteria (Per Phase)
+
+| Phase | Gate |
+|-------|------|
+| Phase 0 complete | Real-world top-3 accuracy ≥ 30% (up from 0%) |
+| Phase 1 complete | `POST /upload` → stored in DB → returns matches |
+| Phase 2 complete | User can upload .sub file in browser, see it on map |
+| Phase 3 complete | ML model reduces "unknown" rate from 45% → < 25% |
+| Phase 4 complete | `docker compose up` → production site live on VPS |
+| Phase 5 complete | 100+ community-verified identifications |
+
+---
+
+*Generated from deep analysis of project state + `docs/ML_DESIGN.md` + `REAL_TEST_RESULTS.md` + `docs/RF_SIGNAL_ANALYSIS_RESEARCH.md` + `docs/FREQUENCY_DEVICE_CHART.md` on 2026-07-17.*
@@ -395,6 +395,13 @@ async def upload_captures(
        sub_parser = SubFileParser()
        gps_extractor = GPSFilenameExtractor()

+        # Map manifest entries by filename so each file gets its own GPS/timestamp
+        # (previously every file was assigned captures[0], breaking multi-file uploads)
+        manifest_map = {
+            c.get("filename"): c
+            for c in manifest_data.get("captures", [])
+        }
+
        successful = []
        failed = []

@@ -429,6 +436,18 @@ async def upload_captures(
                # Get best match from unified results
                best_match = match_results[0] if match_results else None

+                # Per-file manifest entry (falls back to first entry, then empty)
+                entry = manifest_map.get(file.filename)
+                if entry is None:
+                    entry = manifest_data.get("captures", [{}])[0]
+
+                # The real device category comes from the category router (e.g.
+                # "Weather Sensor", "Remote Control"), carried in match_details.
+                best_category = (
+                    best_match.match_details.get("predicted_category")
+                    if best_match else None
+                )
+
                # Use GPS from manifest or filename
                global upload_counter
                upload_counter += 1
@@ -439,15 +458,15 @@ async def upload_captures(
                    "frequency": frequency,
                    "protocol": protocol,
                    "preset": preset,
-                    "latitude": gps_coords.latitude if gps_coords else manifest_data.get("captures", [{}])[0].get("latitude"),
-                    "longitude": gps_coords.longitude if gps_coords else manifest_data.get("captures", [{}])[0].get("longitude"),
+                    "latitude": gps_coords.latitude if gps_coords else entry.get("latitude"),
+                    "longitude": gps_coords.longitude if gps_coords else entry.get("longitude"),
                    "gps_source": gps_coords.source if gps_coords else "manual",
-                    "timestamp": manifest_data.get("captures", [{}])[0].get("timestamp", ""),
+                    "timestamp": entry.get("timestamp", ""),
                    "data_source": manifest_data.get("data_source", "production"),  # production, test, or mock
                    "session_id": manifest_data.get("session_uuid", ""),
                    # Device identification fields (from unified matcher)
                    "device_name": best_match.device_name if best_match else None,
-                    "device_category": f"{best_match.manufacturer} - {best_match.device_name}" if best_match else None,
+                    "device_category": best_category,
                    "match_confidence": best_match.confidence if best_match else None,
                    "match_method": best_match.match_method if best_match else None,
                    "device_description": f"{best_match.manufacturer} {best_match.device_name}" if best_match else None,
@@ -456,6 +475,7 @@ async def upload_captures(
                        {
                            "device_name": m.device_name,
                            "manufacturer": m.manufacturer,
+                            "category": m.match_details.get("predicted_category"),
                            "confidence": m.confidence,
                            "method": m.match_method,
                            "details": m.match_details