fix: correct per-file GPS and device category in upload handler

- main_simple.py upload: map manifest entries by filename so each file gets its own GPS/timestamp (previously every file was assigned captures[0], breaking multi-file uploads) - device_category now uses the category router's real category (e.g. "Remote Control") instead of the mislabeled "manufacturer - device_name" string; also surface category per matched device - add PLAN_TO_PROD.md / FABLE.md development briefs - add CONTEXT.md (leaked PAT redacted from remote URL) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-18 13:09:52 -07:00
parent 1b9b33fa3f
commit 78c28fe43f
4 changed files with 888 additions and 4 deletions
@@ -0,0 +1,318 @@
+# GigLez — FABLE Development Brief
+**Project**: Sub-GHz IoT RF device mapping platform (Wigle for IoT)  
+**Repo**: `/home/dell/coding/security/giglez`  
+**Branch**: `p1-p2-validation`  
+**Briefed**: 2026-07-17
+
+---
+
+## Your Job
+
+You are picking up active development on GigLez. Phase 0 is complete. Your job is to drive Phases 1–2 (API completion + web UI) to a state where a user can open a browser, upload a `.sub` file, and see it appear on a map. That is the MVP gate.
+
+Read this brief fully before writing a single line of code. The project has a lot of prior work — most of what you'd think to build already exists in some form. Explore before you scaffold.
+
+---
+
+## Situational Awareness
+
+### What Exists and Works
+
+| Layer | Files | Confidence |
+|-------|-------|-----------|
+| `.sub` / RAW parser | `src/parser/sub_parser.py` (308 LOC) | Production-ready |
+| GPS validator | `src/gps/validator.py` (22/22 tests pass) | Production-ready |
+| Device matcher | `src/matcher/engine.py` → `device_identifier.py` | 67% top-3 real-world |
+| Category router | `src/matcher/category_router.py` | Complete (Phase 0) |
+| Protocol DB | `src/matcher/protocol_database.py` | 350+ protocols |
+| DB schema | `scripts/create_schema.sql` (670 LOC, PostGIS) | Ready to init |
+| ORM models | `src/database/models.py` (11 tables) | Complete |
+| Storage backends | `src/core/storage/` (FS + S3) | Complete |
+| Config system | `config/settings.py` | Dev/prod aware |
+
+### What's Skeleton Only (Needs You)
+
+| Layer | Files | Gap |
+|-------|-------|-----|
+| Upload endpoint | `src/api/routes/captures.py` | 60 LOC of boilerplate, never completed |
+| Search endpoint | `src/api/routes/query.py` | 4-line stub |
+| Stats endpoint | `src/api/routes/stats.py` | 3-line stub |
+| Devices endpoint | `src/api/routes/devices.py` | 5-line stub |
+| FastAPI app | `src/api/main_simple.py` | Uses MockDB, old strategies |
+| Web UI | `templates/`, `static/js/` | Empty structures |
+| Auth | `src/api/dependencies.py` | `optional_auth` stub exists |
+
+### Duplicate/Dead Files — Ignore These
+
+- `src/api/main_orm_legacy.py` — superseded
+- `src/api/routes/captures_enhanced.py` — superseded by `captures.py`
+- `src/api/routes/captures_with_logging.py` — superseded
+- `src/matcher/strategies.py` — legacy fallback, don't extend
+
+---
+
+## Architecture Snapshot
+
+```
+Browser / CLI client
+        │
+        ▼
+FastAPI (src/api/main_simple.py → currently MockDB mode)
+        │
+        ├── POST /api/v1/captures/upload
+        │       → parse_sub_file() → validate_gps() → matcher.match() → db.store()
+        │
+        ├── GET  /api/v1/search?bbox=...&freq=...&category=...
+        │       → PostGIS ST_Within → returns GeoJSON
+        │
+        ├── GET  /api/v1/heatmap
+        │       → materialized view capture_heatmap → grid densities
+        │
+        ├── GET  /api/v1/stats
+        │       → materialized view device_statistics
+        │
+        └── GET  /api/v1/devices
+                → known device catalog
+
+Database: PostgreSQL + PostGIS
+  Schema at: scripts/create_schema.sql
+  Dev: SQLite fallback acceptable for MVP
+  Prod: PostgreSQL required
+
+Storage: src/core/storage/factory.py
+  Dev:  STORAGE_BACKEND=filesystem (local ./data/uploads/)
+  Prod: STORAGE_BACKEND=s3
+```
+
+---
+
+## Phase 1: Complete the Backend API
+
+**Goal**: `POST /api/v1/captures/upload` → stored in DB → returns match results.
+
+### 1.1 Reconcile the FastAPI entry point
+
+`src/api/main_simple.py` currently uses `MockDB` and imports the old strategy classes. You need to:
+- Replace `MockDB` with the real DB session from `config/database.py`
+- Wire in the proper router pattern
+- Make the app runnable in "no-DB" dev mode (SQLite or in-memory) so you don't block on PostgreSQL setup
+- Keep the existing `/docs` and `/redoc` endpoints
+
+**Dev mode fallback**: If `DATABASE_URL` is not set, use `sqlite:///./giglez.db` (already exists at project root from prior testing). The ORM models already use SQLAlchemy 2.0, so this is a connection string swap.
+
+### 1.2 Complete `POST /api/v1/captures/upload`
+
+The skeleton is at `src/api/routes/captures.py:29`. It has the right shape — manifest parsing, file hashing — but never stores or matches. Complete it:
+
+```
+receive multipart (manifest JSON + .sub files)
+    → validate manifest schema (pydantic)
+    → for each file:
+        sha256 = hash(bytes)
+        check duplicate (Capture.file_hash in db) → skip if exists
+        metadata = parse_sub_file(tmp_path)
+        gps = validate_gps_coordinates(lat, lon, accuracy)
+        matches = SignatureMatcher().match(metadata, max_results=5)
+        file_url = storage.save(sha256, bytes)
+        db.add(Capture(...))
+        db.add_all([CaptureMatch(...) for top 3 matches])
+    → return {
+        processed: int,
+        skipped_duplicates: int,
+        captures: [{id, lat, lon, matches: [{device, confidence, category}]}]
+      }
+```
+
+Use `SignatureMatcher` from `src/matcher/engine.py`, not the old strategies.
+
+### 1.3 Complete `GET /api/v1/search`
+
+```python
+@router.get("/search")
+async def search_captures(
+    lat: float, lon: float, radius_km: float = 1.0,
+    frequency_min: Optional[int] = None,
+    frequency_max: Optional[int] = None,
+    category: Optional[str] = None,
+    limit: int = 100,
+    db: Session = Depends(get_db),
+):
+    # PostGIS: ST_DWithin(geometry, ST_SetSRID(ST_MakePoint(lon, lat), 4326), radius_m)
+    # Return GeoJSON FeatureCollection for Leaflet
+```
+
+For the SQLite dev fallback, do a simple lat/lon bounding box (haversine already in `src/gps/validator.py`).
+
+### 1.4 Complete `GET /api/v1/stats` and `GET /api/v1/devices`
+
+Stats: total captures, unique device types, geographic coverage (distinct countries/cities), top 5 protocols.  
+Devices: paginated list of known device types with capture counts.
+
+### 1.5 Auth middleware
+
+`src/api/dependencies.py` has `optional_auth` stub. For MVP:
+- Anonymous uploads: allowed, rate-limited by IP (use `slowapi` or simple in-memory counter)
+- API key (optional): `X-API-Key` header, stored in `users.api_key` column
+- No JWT yet — defer to Phase 5
+
+---
+
+## Phase 2: Web UI MVP
+
+**Goal**: User opens browser → sees map → drags `.sub` file → sees it appear as a marker.
+
+Approach: Jinja2 templates + Leaflet.js via CDN. No build step. No framework. Vanilla JS.
+
+### 2.1 Templates needed
+
+```
+templates/
+├── base.html       ← nav, CDN links (Leaflet 1.9, Tailwind CDN, htmx 1.9)
+├── index.html      ← landing: stats summary + recent 10 captures on mini-map
+├── map.html        ← full-screen Leaflet map + filter sidebar
+└── upload.html     ← drag-and-drop upload form
+```
+
+Use **htmx** for the upload form — it makes multipart upload + response handling trivial without writing JS. The upload endpoint returns partial HTML with match results that htmx swaps in.
+
+### 2.2 Leaflet map (map.html)
+
+```javascript
+const map = L.map('map').setView([39.5, -98.35], 4);  // USA center
+L.tileLayer('https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png').addTo(map);
+
+// Marker cluster
+const markers = L.markerClusterGroup();
+
+// Load from API
+fetch('/api/v1/search?lat=39.5&lon=-98.35&radius_km=5000')
+  .then(r => r.json())
+  .then(geojson => {
+    L.geoJSON(geojson, {
+      onEachFeature: (feature, layer) => {
+        layer.bindPopup(`
+          <b>${feature.properties.device_name}</b><br>
+          ${feature.properties.frequency_mhz} MHz<br>
+          ${feature.properties.category}<br>
+          Confidence: ${feature.properties.confidence}%
+        `);
+      }
+    }).addTo(markers);
+    map.addLayer(markers);
+  });
+```
+
+CDN links needed:
+- `https://unpkg.com/leaflet@1.9.4/dist/leaflet.css`
+- `https://unpkg.com/leaflet@1.9.4/dist/leaflet.js`
+- `https://unpkg.com/leaflet.markercluster@1.5.3/dist/MarkerCluster.css`
+- `https://unpkg.com/leaflet.markercluster@1.5.3/dist/leaflet.markercluster.js`
+- `https://cdn.tailwindcss.com`
+- `https://unpkg.com/htmx.org@1.9.10`
+
+### 2.3 Upload form (upload.html)
+
+```html
+<form hx-post="/api/v1/captures/upload"
+      hx-encoding="multipart/form-data"
+      hx-target="#results"
+      hx-swap="innerHTML">
+  <input type="hidden" name="manifest" id="manifest-json">
+  <input type="file" name="files" multiple accept=".sub">
+  <!-- GPS inputs with "Use my location" button -->
+  <input name="latitude" placeholder="Latitude">
+  <input name="longitude" placeholder="Longitude">
+  <button>Upload</button>
+</form>
+<div id="results"></div>
+```
+
+The upload endpoint should return an HTML partial (not JSON) when `Accept: text/html` — show matches as cards.
+
+---
+
+## Key Technical Constraints
+
+1. **Don't touch `src/matcher/`** except to import from it. The Phase 0 accuracy work is complete. The only valid change is importing the new `SignatureMatcher` in the API.
+
+2. **Don't add Redis yet**. Rate limiting can be in-memory dict for MVP. Redis is Phase 4.
+
+3. **No JWT yet**. API key + anonymous is sufficient for MVP.
+
+4. **SQLite dev mode is mandatory**. Not everyone has PostgreSQL running. The app must start with `uvicorn src.api.main:app --reload` and work without any external services.
+
+5. **Don't create `requirements.txt` additions unless strictly necessary**. Everything needed is already installed: fastapi, uvicorn, sqlalchemy, jinja2, pydantic, loguru, numpy, scipy, scikit-learn, geopy.
+
+6. **`htmx`** for the upload form only — don't add it to the map page. Leaflet is sufficient there.
+
+---
+
+## How to Run / Test
+
+```bash
+# From project root
+cd /home/dell/coding/security/giglez
+
+# Dev server (SQLite, no DB setup needed)
+DATABASE_URL=sqlite:///./giglez.db uvicorn src.api.main:app --reload
+
+# Run existing tests
+python3 -m pytest tests/unit/ -q
+
+# Phase 0 accuracy benchmark
+python3 scripts/benchmark_phase0.py
+
+# Quick matcher smoke test
+python3 -c "
+from src.matcher.engine import SignatureMatcher
+from src.parser.sub_parser import parse_sub_file
+# ... test with a .sub file from signatures/t-embed-rf/
+"
+```
+
+---
+
+## How to Use Your Capabilities Here
+
+You are FABLE — Opus 4.6, the most capable model in the Claude lineup. Use that accordingly:
+
+**Parallelize ruthlessly.** Phase 1 API routes and Phase 2 templates are independent. Use the Agent tool to run them simultaneously: one subagent completing all 4 API routes while another writes the 4 templates. You should be orchestrating, not writing every line sequentially yourself.
+
+**Make architectural calls, don't ask.** If you see a better pattern than what's described here — use it. The constraints above are floors, not ceilings. If you see a cleaner way to structure the upload response, ship it.
+
+**Read before writing.** Before touching any file, read it. The project has multiple versions of things (captures.py, captures_enhanced.py, captures_with_logging.py). The right one is `captures.py`. Read the 60 existing lines before adding to them.
+
+**Fix the right thing.** The upload endpoint is the #1 priority. If you complete nothing else, complete `POST /api/v1/captures/upload` end-to-end with a working SQLite fallback. A user being able to upload and get match results back is the first tangible milestone.
+
+**The map is the moat.** The moment captures appear as markers on a Leaflet map is when GigLez becomes a real product rather than a Python library. Prioritize that moment.
+
+---
+
+## Success Criteria
+
+| Milestone | Criteria |
+|-----------|----------|
+| Phase 1 done | `curl -X POST .../captures/upload -F manifest=... -F files=@test.sub` returns match JSON |
+| Phase 2 done | `http://localhost:8000/map` shows markers for uploaded captures |
+| MVP gate | Both above, running on SQLite, no PostgreSQL required |
+
+---
+
+## Related Files to Read First
+
+```
+PLAN_TO_PROD.md              ← Full production roadmap
+REAL_TEST_RESULTS.md         ← Baseline accuracy data (before Phase 0)
+src/api/main_simple.py       ← Current FastAPI entry point
+src/api/routes/captures.py   ← Upload endpoint skeleton
+src/api/dependencies.py      ← Auth + DB dependency stubs
+src/database/models.py       ← All ORM models
+config/settings.py           ← Environment-aware settings
+config/database.py           ← DB connection management
+scripts/create_schema.sql    ← PostgreSQL schema (reference)
+.env.development             ← Dev environment template
+```
+
+---
+
+*Last updated: 2026-07-17 after Phase 0 completion (0% → 67% top-3 real-world accuracy).*