317 lines
12 KiB
Markdown
317 lines
12 KiB
Markdown
# GigLez — FABLE Development Brief
|
||
**Project**: Sub-GHz IoT RF device mapping platform (Wigle for IoT)
|
||
**Briefed**: 2026-07-17
|
||
|
||
---
|
||
|
||
## Your Job
|
||
|
||
You are picking up active development on GigLez. Phase 0 is complete. Your job is to drive Phases 1–2 (API completion + web UI) to a state where a user can open a browser, upload a `.sub` file, and see it appear on a map. That is the MVP gate.
|
||
|
||
Read this brief fully before writing a single line of code. The project has a lot of prior work — most of what you'd think to build already exists in some form. Explore before you scaffold.
|
||
|
||
---
|
||
|
||
## Situational Awareness
|
||
|
||
### What Exists and Works
|
||
|
||
| Layer | Files | Confidence |
|
||
|-------|-------|-----------|
|
||
| `.sub` / RAW parser | `src/parser/sub_parser.py` (308 LOC) | Production-ready |
|
||
| GPS validator | `src/gps/validator.py` (22/22 tests pass) | Production-ready |
|
||
| Device matcher | `src/matcher/engine.py` → `device_identifier.py` | 67% top-3 real-world |
|
||
| Category router | `src/matcher/category_router.py` | Complete (Phase 0) |
|
||
| Protocol DB | `src/matcher/protocol_database.py` | 350+ protocols |
|
||
| DB schema | `scripts/create_schema.sql` (670 LOC, PostGIS) | Ready to init |
|
||
| ORM models | `src/database/models.py` (11 tables) | Complete |
|
||
| Storage backends | `src/core/storage/` (FS + S3) | Complete |
|
||
| Config system | `config/settings.py` | Dev/prod aware |
|
||
|
||
### What's Skeleton Only (Needs You)
|
||
|
||
| Layer | Files | Gap |
|
||
|-------|-------|-----|
|
||
| Upload endpoint | `src/api/routes/captures.py` | 60 LOC of boilerplate, never completed |
|
||
| Search endpoint | `src/api/routes/query.py` | 4-line stub |
|
||
| Stats endpoint | `src/api/routes/stats.py` | 3-line stub |
|
||
| Devices endpoint | `src/api/routes/devices.py` | 5-line stub |
|
||
| FastAPI app | `src/api/main_simple.py` | Uses MockDB, old strategies |
|
||
| Web UI | `templates/`, `static/js/` | Empty structures |
|
||
| Auth | `src/api/dependencies.py` | `optional_auth` stub exists |
|
||
|
||
### Duplicate/Dead Files — Ignore These
|
||
|
||
- `src/api/main_orm_legacy.py` — superseded
|
||
- `src/api/routes/captures_enhanced.py` — superseded by `captures.py`
|
||
- `src/api/routes/captures_with_logging.py` — superseded
|
||
- `src/matcher/strategies.py` — legacy fallback, don't extend
|
||
|
||
---
|
||
|
||
## Architecture Snapshot
|
||
|
||
```
|
||
Browser / CLI client
|
||
│
|
||
▼
|
||
FastAPI (src/api/main_simple.py → currently MockDB mode)
|
||
│
|
||
├── POST /api/v1/captures/upload
|
||
│ → parse_sub_file() → validate_gps() → matcher.match() → db.store()
|
||
│
|
||
├── GET /api/v1/search?bbox=...&freq=...&category=...
|
||
│ → PostGIS ST_Within → returns GeoJSON
|
||
│
|
||
├── GET /api/v1/heatmap
|
||
│ → materialized view capture_heatmap → grid densities
|
||
│
|
||
├── GET /api/v1/stats
|
||
│ → materialized view device_statistics
|
||
│
|
||
└── GET /api/v1/devices
|
||
→ known device catalog
|
||
|
||
Database: PostgreSQL + PostGIS
|
||
Schema at: scripts/create_schema.sql
|
||
Dev: SQLite fallback acceptable for MVP
|
||
Prod: PostgreSQL required
|
||
|
||
Storage: src/core/storage/factory.py
|
||
Dev: STORAGE_BACKEND=filesystem (local ./data/uploads/)
|
||
Prod: STORAGE_BACKEND=s3
|
||
```
|
||
|
||
---
|
||
|
||
## Phase 1: Complete the Backend API
|
||
|
||
**Goal**: `POST /api/v1/captures/upload` → stored in DB → returns match results.
|
||
|
||
### 1.1 Reconcile the FastAPI entry point
|
||
|
||
`src/api/main_simple.py` currently uses `MockDB` and imports the old strategy classes. You need to:
|
||
- Replace `MockDB` with the real DB session from `config/database.py`
|
||
- Wire in the proper router pattern
|
||
- Make the app runnable in "no-DB" dev mode (SQLite or in-memory) so you don't block on PostgreSQL setup
|
||
- Keep the existing `/docs` and `/redoc` endpoints
|
||
|
||
**Dev mode fallback**: If `DATABASE_URL` is not set, use `sqlite:///./giglez.db` (already exists at project root from prior testing). The ORM models already use SQLAlchemy 2.0, so this is a connection string swap.
|
||
|
||
### 1.2 Complete `POST /api/v1/captures/upload`
|
||
|
||
The skeleton is at `src/api/routes/captures.py:29`. It has the right shape — manifest parsing, file hashing — but never stores or matches. Complete it:
|
||
|
||
```
|
||
receive multipart (manifest JSON + .sub files)
|
||
→ validate manifest schema (pydantic)
|
||
→ for each file:
|
||
sha256 = hash(bytes)
|
||
check duplicate (Capture.file_hash in db) → skip if exists
|
||
metadata = parse_sub_file(tmp_path)
|
||
gps = validate_gps_coordinates(lat, lon, accuracy)
|
||
matches = SignatureMatcher().match(metadata, max_results=5)
|
||
file_url = storage.save(sha256, bytes)
|
||
db.add(Capture(...))
|
||
db.add_all([CaptureMatch(...) for top 3 matches])
|
||
→ return {
|
||
processed: int,
|
||
skipped_duplicates: int,
|
||
captures: [{id, lat, lon, matches: [{device, confidence, category}]}]
|
||
}
|
||
```
|
||
|
||
Use `SignatureMatcher` from `src/matcher/engine.py`, not the old strategies.
|
||
|
||
### 1.3 Complete `GET /api/v1/search`
|
||
|
||
```python
|
||
@router.get("/search")
|
||
async def search_captures(
|
||
lat: float, lon: float, radius_km: float = 1.0,
|
||
frequency_min: Optional[int] = None,
|
||
frequency_max: Optional[int] = None,
|
||
category: Optional[str] = None,
|
||
limit: int = 100,
|
||
db: Session = Depends(get_db),
|
||
):
|
||
# PostGIS: ST_DWithin(geometry, ST_SetSRID(ST_MakePoint(lon, lat), 4326), radius_m)
|
||
# Return GeoJSON FeatureCollection for Leaflet
|
||
```
|
||
|
||
For the SQLite dev fallback, do a simple lat/lon bounding box (haversine already in `src/gps/validator.py`).
|
||
|
||
### 1.4 Complete `GET /api/v1/stats` and `GET /api/v1/devices`
|
||
|
||
Stats: total captures, unique device types, geographic coverage (distinct countries/cities), top 5 protocols.
|
||
Devices: paginated list of known device types with capture counts.
|
||
|
||
### 1.5 Auth middleware
|
||
|
||
`src/api/dependencies.py` has `optional_auth` stub. For MVP:
|
||
- Anonymous uploads: allowed, rate-limited by IP (use `slowapi` or simple in-memory counter)
|
||
- API key (optional): `X-API-Key` header, stored in `users.api_key` column
|
||
- No JWT yet — defer to Phase 5
|
||
|
||
---
|
||
|
||
## Phase 2: Web UI MVP
|
||
|
||
**Goal**: User opens browser → sees map → drags `.sub` file → sees it appear as a marker.
|
||
|
||
Approach: Jinja2 templates + Leaflet.js via CDN. No build step. No framework. Vanilla JS.
|
||
|
||
### 2.1 Templates needed
|
||
|
||
```
|
||
templates/
|
||
├── base.html ← nav, CDN links (Leaflet 1.9, Tailwind CDN, htmx 1.9)
|
||
├── index.html ← landing: stats summary + recent 10 captures on mini-map
|
||
├── map.html ← full-screen Leaflet map + filter sidebar
|
||
└── upload.html ← drag-and-drop upload form
|
||
```
|
||
|
||
Use **htmx** for the upload form — it makes multipart upload + response handling trivial without writing JS. The upload endpoint returns partial HTML with match results that htmx swaps in.
|
||
|
||
### 2.2 Leaflet map (map.html)
|
||
|
||
```javascript
|
||
const map = L.map('map').setView([39.5, -98.35], 4); // USA center
|
||
L.tileLayer('https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png').addTo(map);
|
||
|
||
// Marker cluster
|
||
const markers = L.markerClusterGroup();
|
||
|
||
// Load from API
|
||
fetch('/api/v1/search?lat=39.5&lon=-98.35&radius_km=5000')
|
||
.then(r => r.json())
|
||
.then(geojson => {
|
||
L.geoJSON(geojson, {
|
||
onEachFeature: (feature, layer) => {
|
||
layer.bindPopup(`
|
||
<b>${feature.properties.device_name}</b><br>
|
||
${feature.properties.frequency_mhz} MHz<br>
|
||
${feature.properties.category}<br>
|
||
Confidence: ${feature.properties.confidence}%
|
||
`);
|
||
}
|
||
}).addTo(markers);
|
||
map.addLayer(markers);
|
||
});
|
||
```
|
||
|
||
CDN links needed:
|
||
- `https://unpkg.com/leaflet@1.9.4/dist/leaflet.css`
|
||
- `https://unpkg.com/leaflet@1.9.4/dist/leaflet.js`
|
||
- `https://unpkg.com/leaflet.markercluster@1.5.3/dist/MarkerCluster.css`
|
||
- `https://unpkg.com/leaflet.markercluster@1.5.3/dist/leaflet.markercluster.js`
|
||
- `https://cdn.tailwindcss.com`
|
||
- `https://unpkg.com/htmx.org@1.9.10`
|
||
|
||
### 2.3 Upload form (upload.html)
|
||
|
||
```html
|
||
<form hx-post="/api/v1/captures/upload"
|
||
hx-encoding="multipart/form-data"
|
||
hx-target="#results"
|
||
hx-swap="innerHTML">
|
||
<input type="hidden" name="manifest" id="manifest-json">
|
||
<input type="file" name="files" multiple accept=".sub">
|
||
<!-- GPS inputs with "Use my location" button -->
|
||
<input name="latitude" placeholder="Latitude">
|
||
<input name="longitude" placeholder="Longitude">
|
||
<button>Upload</button>
|
||
</form>
|
||
<div id="results"></div>
|
||
```
|
||
|
||
The upload endpoint should return an HTML partial (not JSON) when `Accept: text/html` — show matches as cards.
|
||
|
||
---
|
||
|
||
## Key Technical Constraints
|
||
|
||
1. **Don't touch `src/matcher/`** except to import from it. The Phase 0 accuracy work is complete. The only valid change is importing the new `SignatureMatcher` in the API.
|
||
|
||
2. **Don't add Redis yet**. Rate limiting can be in-memory dict for MVP. Redis is Phase 4.
|
||
|
||
3. **No JWT yet**. API key + anonymous is sufficient for MVP.
|
||
|
||
4. **SQLite dev mode is mandatory**. Not everyone has PostgreSQL running. The app must start with `uvicorn src.api.main:app --reload` and work without any external services.
|
||
|
||
5. **Don't create `requirements.txt` additions unless strictly necessary**. Everything needed is already installed: fastapi, uvicorn, sqlalchemy, jinja2, pydantic, loguru, numpy, scipy, scikit-learn, geopy.
|
||
|
||
6. **`htmx`** for the upload form only — don't add it to the map page. Leaflet is sufficient there.
|
||
|
||
---
|
||
|
||
## How to Run / Test
|
||
|
||
```bash
|
||
# From project root
|
||
cd /home/dell/coding/security/giglez
|
||
|
||
# Dev server (SQLite, no DB setup needed)
|
||
DATABASE_URL=sqlite:///./giglez.db uvicorn src.api.main:app --reload
|
||
|
||
# Run existing tests
|
||
python3 -m pytest tests/unit/ -q
|
||
|
||
# Phase 0 accuracy benchmark
|
||
python3 scripts/benchmark_phase0.py
|
||
|
||
# Quick matcher smoke test
|
||
python3 -c "
|
||
from src.matcher.engine import SignatureMatcher
|
||
from src.parser.sub_parser import parse_sub_file
|
||
# ... test with a .sub file from signatures/t-embed-rf/
|
||
"
|
||
```
|
||
|
||
---
|
||
|
||
## How to Use Your Capabilities Here
|
||
|
||
You are FABLE — Opus 4.6, the most capable model in the Claude lineup. Use that accordingly:
|
||
|
||
**Parallelize ruthlessly.** Phase 1 API routes and Phase 2 templates are independent. Use the Agent tool to run them simultaneously: one subagent completing all 4 API routes while another writes the 4 templates. You should be orchestrating, not writing every line sequentially yourself.
|
||
|
||
**Make architectural calls, don't ask.** If you see a better pattern than what's described here — use it. The constraints above are floors, not ceilings. If you see a cleaner way to structure the upload response, ship it.
|
||
|
||
**Read before writing.** Before touching any file, read it. The project has multiple versions of things (captures.py, captures_enhanced.py, captures_with_logging.py). The right one is `captures.py`. Read the 60 existing lines before adding to them.
|
||
|
||
**Fix the right thing.** The upload endpoint is the #1 priority. If you complete nothing else, complete `POST /api/v1/captures/upload` end-to-end with a working SQLite fallback. A user being able to upload and get match results back is the first tangible milestone.
|
||
|
||
**The map is the moat.** The moment captures appear as markers on a Leaflet map is when GigLez becomes a real product rather than a Python library. Prioritize that moment.
|
||
|
||
---
|
||
|
||
## Success Criteria
|
||
|
||
| Milestone | Criteria |
|
||
|-----------|----------|
|
||
| Phase 1 done | `curl -X POST .../captures/upload -F manifest=... -F files=@test.sub` returns match JSON |
|
||
| Phase 2 done | `http://localhost:8000/map` shows markers for uploaded captures |
|
||
| MVP gate | Both above, running on SQLite, no PostgreSQL required |
|
||
|
||
---
|
||
|
||
## Related Files to Read First
|
||
|
||
```
|
||
PLAN_TO_PROD.md ← Full production roadmap
|
||
REAL_TEST_RESULTS.md ← Baseline accuracy data (before Phase 0)
|
||
src/api/main_simple.py ← Current FastAPI entry point
|
||
src/api/routes/captures.py ← Upload endpoint skeleton
|
||
src/api/dependencies.py ← Auth + DB dependency stubs
|
||
src/database/models.py ← All ORM models
|
||
config/settings.py ← Environment-aware settings
|
||
config/database.py ← DB connection management
|
||
scripts/create_schema.sql ← PostgreSQL schema (reference)
|
||
.env.development ← Dev environment template
|
||
```
|
||
|
||
---
|
||
|
||
*Last updated: 2026-07-17 after Phase 0 completion (0% → 67% top-3 real-world accuracy).*
|