Trilltechnician/giglez

Fork 0

Files

T

Trilltechnician 643f5ac74b Update FABLE.md

2026-07-19 17:00:24 +00:00

12 KiB

Raw Permalink Blame History

GigLez — FABLE Development Brief

Project: Sub-GHz IoT RF device mapping platform (Wigle for IoT)
Briefed: 2026-07-17

Your Job

You are picking up active development on GigLez. Phase 0 is complete. Your job is to drive Phases 1–2 (API completion + web UI) to a state where a user can open a browser, upload a .sub file, and see it appear on a map. That is the MVP gate.

Read this brief fully before writing a single line of code. The project has a lot of prior work — most of what you'd think to build already exists in some form. Explore before you scaffold.

Situational Awareness

What Exists and Works

Layer	Files	Confidence
`.sub` / RAW parser	`src/parser/sub_parser.py` (308 LOC)	Production-ready
GPS validator	`src/gps/validator.py` (22/22 tests pass)	Production-ready
Device matcher	`src/matcher/engine.py` → `device_identifier.py`	67% top-3 real-world
Category router	`src/matcher/category_router.py`	Complete (Phase 0)
Protocol DB	`src/matcher/protocol_database.py`	350+ protocols
DB schema	`scripts/create_schema.sql` (670 LOC, PostGIS)	Ready to init
ORM models	`src/database/models.py` (11 tables)	Complete
Storage backends	`src/core/storage/` (FS + S3)	Complete
Config system	`config/settings.py`	Dev/prod aware

What's Skeleton Only (Needs You)

Layer	Files	Gap
Upload endpoint	`src/api/routes/captures.py`	60 LOC of boilerplate, never completed
Search endpoint	`src/api/routes/query.py`	4-line stub
Stats endpoint	`src/api/routes/stats.py`	3-line stub
Devices endpoint	`src/api/routes/devices.py`	5-line stub
FastAPI app	`src/api/main_simple.py`	Uses MockDB, old strategies
Web UI	`templates/`, `static/js/`	Empty structures
Auth	`src/api/dependencies.py`	`optional_auth` stub exists

Duplicate/Dead Files — Ignore These

src/api/main_orm_legacy.py — superseded
src/api/routes/captures_enhanced.py — superseded by captures.py
src/api/routes/captures_with_logging.py — superseded
src/matcher/strategies.py — legacy fallback, don't extend

Architecture Snapshot

Browser / CLI client
        │
        ▼
FastAPI (src/api/main_simple.py → currently MockDB mode)
        │
        ├── POST /api/v1/captures/upload
        │       → parse_sub_file() → validate_gps() → matcher.match() → db.store()
        │
        ├── GET  /api/v1/search?bbox=...&freq=...&category=...
        │       → PostGIS ST_Within → returns GeoJSON
        │
        ├── GET  /api/v1/heatmap
        │       → materialized view capture_heatmap → grid densities
        │
        ├── GET  /api/v1/stats
        │       → materialized view device_statistics
        │
        └── GET  /api/v1/devices
                → known device catalog

Database: PostgreSQL + PostGIS
  Schema at: scripts/create_schema.sql
  Dev: SQLite fallback acceptable for MVP
  Prod: PostgreSQL required

Storage: src/core/storage/factory.py
  Dev:  STORAGE_BACKEND=filesystem (local ./data/uploads/)
  Prod: STORAGE_BACKEND=s3

Phase 1: Complete the Backend API

Goal: POST /api/v1/captures/upload → stored in DB → returns match results.

1.1 Reconcile the FastAPI entry point

src/api/main_simple.py currently uses MockDB and imports the old strategy classes. You need to:

Replace MockDB with the real DB session from config/database.py
Wire in the proper router pattern
Make the app runnable in "no-DB" dev mode (SQLite or in-memory) so you don't block on PostgreSQL setup
Keep the existing /docs and /redoc endpoints

Dev mode fallback: If DATABASE_URL is not set, use sqlite:///./giglez.db (already exists at project root from prior testing). The ORM models already use SQLAlchemy 2.0, so this is a connection string swap.

1.2 Complete `POST /api/v1/captures/upload`

The skeleton is at src/api/routes/captures.py:29. It has the right shape — manifest parsing, file hashing — but never stores or matches. Complete it:

receive multipart (manifest JSON + .sub files)
    → validate manifest schema (pydantic)
    → for each file:
        sha256 = hash(bytes)
        check duplicate (Capture.file_hash in db) → skip if exists
        metadata = parse_sub_file(tmp_path)
        gps = validate_gps_coordinates(lat, lon, accuracy)
        matches = SignatureMatcher().match(metadata, max_results=5)
        file_url = storage.save(sha256, bytes)
        db.add(Capture(...))
        db.add_all([CaptureMatch(...) for top 3 matches])
    → return {
        processed: int,
        skipped_duplicates: int,
        captures: [{id, lat, lon, matches: [{device, confidence, category}]}]
      }

Use SignatureMatcher from src/matcher/engine.py, not the old strategies.

1.3 Complete `GET /api/v1/search`

@router.get("/search")
async def search_captures(
    lat: float, lon: float, radius_km: float = 1.0,
    frequency_min: Optional[int] = None,
    frequency_max: Optional[int] = None,
    category: Optional[str] = None,
    limit: int = 100,
    db: Session = Depends(get_db),
):
    # PostGIS: ST_DWithin(geometry, ST_SetSRID(ST_MakePoint(lon, lat), 4326), radius_m)
    # Return GeoJSON FeatureCollection for Leaflet

For the SQLite dev fallback, do a simple lat/lon bounding box (haversine already in src/gps/validator.py).

1.4 Complete `GET /api/v1/stats` and `GET /api/v1/devices`

Stats: total captures, unique device types, geographic coverage (distinct countries/cities), top 5 protocols.
Devices: paginated list of known device types with capture counts.

1.5 Auth middleware

src/api/dependencies.py has optional_auth stub. For MVP:

Anonymous uploads: allowed, rate-limited by IP (use slowapi or simple in-memory counter)
API key (optional): X-API-Key header, stored in users.api_key column
No JWT yet — defer to Phase 5

Phase 2: Web UI MVP

Goal: User opens browser → sees map → drags .sub file → sees it appear as a marker.

Approach: Jinja2 templates + Leaflet.js via CDN. No build step. No framework. Vanilla JS.

2.1 Templates needed

templates/
├── base.html       ← nav, CDN links (Leaflet 1.9, Tailwind CDN, htmx 1.9)
├── index.html      ← landing: stats summary + recent 10 captures on mini-map
├── map.html        ← full-screen Leaflet map + filter sidebar
└── upload.html     ← drag-and-drop upload form

Use htmx for the upload form — it makes multipart upload + response handling trivial without writing JS. The upload endpoint returns partial HTML with match results that htmx swaps in.

2.2 Leaflet map (map.html)

const map = L.map('map').setView([39.5, -98.35], 4);  // USA center
L.tileLayer('https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png').addTo(map);

// Marker cluster
const markers = L.markerClusterGroup();

// Load from API
fetch('/api/v1/search?lat=39.5&lon=-98.35&radius_km=5000')
  .then(r => r.json())
  .then(geojson => {
    L.geoJSON(geojson, {
      onEachFeature: (feature, layer) => {
        layer.bindPopup(`
          <b>${feature.properties.device_name}</b><br>
          ${feature.properties.frequency_mhz} MHz<br>
          ${feature.properties.category}<br>
          Confidence: ${feature.properties.confidence}%
        `);
      }
    }).addTo(markers);
    map.addLayer(markers);
  });

CDN links needed:

https://unpkg.com/leaflet@1.9.4/dist/leaflet.css
https://unpkg.com/leaflet@1.9.4/dist/leaflet.js
https://unpkg.com/leaflet.markercluster@1.5.3/dist/MarkerCluster.css
https://unpkg.com/leaflet.markercluster@1.5.3/dist/leaflet.markercluster.js
https://cdn.tailwindcss.com
https://unpkg.com/htmx.org@1.9.10

2.3 Upload form (upload.html)

<form hx-post="/api/v1/captures/upload"
      hx-encoding="multipart/form-data"
      hx-target="#results"
      hx-swap="innerHTML">
  <input type="hidden" name="manifest" id="manifest-json">
  <input type="file" name="files" multiple accept=".sub">
  <!-- GPS inputs with "Use my location" button -->
  <input name="latitude" placeholder="Latitude">
  <input name="longitude" placeholder="Longitude">
  <button>Upload</button>
</form>
<div id="results"></div>

The upload endpoint should return an HTML partial (not JSON) when Accept: text/html — show matches as cards.

Key Technical Constraints

Don't touch src/matcher/ except to import from it. The Phase 0 accuracy work is complete. The only valid change is importing the new SignatureMatcher in the API.
Don't add Redis yet. Rate limiting can be in-memory dict for MVP. Redis is Phase 4.
No JWT yet. API key + anonymous is sufficient for MVP.
SQLite dev mode is mandatory. Not everyone has PostgreSQL running. The app must start with uvicorn src.api.main:app --reload and work without any external services.
Don't create requirements.txt additions unless strictly necessary. Everything needed is already installed: fastapi, uvicorn, sqlalchemy, jinja2, pydantic, loguru, numpy, scipy, scikit-learn, geopy.
htmx for the upload form only — don't add it to the map page. Leaflet is sufficient there.

How to Run / Test

# From project root
cd /home/dell/coding/security/giglez

# Dev server (SQLite, no DB setup needed)
DATABASE_URL=sqlite:///./giglez.db uvicorn src.api.main:app --reload

# Run existing tests
python3 -m pytest tests/unit/ -q

# Phase 0 accuracy benchmark
python3 scripts/benchmark_phase0.py

# Quick matcher smoke test
python3 -c "
from src.matcher.engine import SignatureMatcher
from src.parser.sub_parser import parse_sub_file
# ... test with a .sub file from signatures/t-embed-rf/
"

How to Use Your Capabilities Here

You are FABLE — Opus 4.6, the most capable model in the Claude lineup. Use that accordingly:

Parallelize ruthlessly. Phase 1 API routes and Phase 2 templates are independent. Use the Agent tool to run them simultaneously: one subagent completing all 4 API routes while another writes the 4 templates. You should be orchestrating, not writing every line sequentially yourself.

Make architectural calls, don't ask. If you see a better pattern than what's described here — use it. The constraints above are floors, not ceilings. If you see a cleaner way to structure the upload response, ship it.

Read before writing. Before touching any file, read it. The project has multiple versions of things (captures.py, captures_enhanced.py, captures_with_logging.py). The right one is captures.py. Read the 60 existing lines before adding to them.

Fix the right thing. The upload endpoint is the #1 priority. If you complete nothing else, complete POST /api/v1/captures/upload end-to-end with a working SQLite fallback. A user being able to upload and get match results back is the first tangible milestone.

The map is the moat. The moment captures appear as markers on a Leaflet map is when GigLez becomes a real product rather than a Python library. Prioritize that moment.

Success Criteria

Milestone	Criteria
Phase 1 done	`curl -X POST .../captures/upload -F manifest=... -F files=@test.sub` returns match JSON
Phase 2 done	`http://localhost:8000/map` shows markers for uploaded captures
MVP gate	Both above, running on SQLite, no PostgreSQL required

PLAN_TO_PROD.md              ← Full production roadmap
REAL_TEST_RESULTS.md         ← Baseline accuracy data (before Phase 0)
src/api/main_simple.py       ← Current FastAPI entry point
src/api/routes/captures.py   ← Upload endpoint skeleton
src/api/dependencies.py      ← Auth + DB dependency stubs
src/database/models.py       ← All ORM models
config/settings.py           ← Environment-aware settings
config/database.py           ← DB connection management
scripts/create_schema.sql    ← PostgreSQL schema (reference)
.env.development             ← Dev environment template

Last updated: 2026-07-17 after Phase 0 completion (0% → 67% top-3 real-world accuracy).

12 KiB Raw Permalink Blame History Unescape Escape