From 58ba021de12ad31306cd50dad968cbe506177104 Mon Sep 17 00:00:00 2001 From: leetcrypt Date: Mon, 20 Jul 2026 22:30:30 -0700 Subject: [PATCH] test: add end-to-end UI verification harness for deploy artifact Adds VERIFICATION.md + scripts/verify_ui.mjs, a Playwright-driven check of the full user journey against the deployed SPA: page loads clean, Leaflet map renders, a real DOM upload (.sub + GPS) returns a device match, and the capture store increments. Grounded in real element IDs and live endpoints. In-sandbox this session covered the code-correctness layers (L1-substance parse->match->category, L3 save/load round-trip, L4 BinRAW + per-device catalog category regressions). The runtime/transport layers still require a live listening server and must be run from a normal shell: - L0 docker compose up --build, container healthy - L1 HTTP curl/jq /health + static assets 200 - L2 Playwright browser journey (scripts/verify_ui.mjs) - L3 restart container up->down->up, capture count unchanged (volume) Also ignores trained model binaries (models/*.onnx, *.pt) and the verify-shots/ screenshot output. Co-Authored-By: Claude Opus 4.6 --- .gitignore | 7 + VERIFICATION.md | 304 ++++++++++++++++++++++++++++++++++++++++++ scripts/verify_ui.mjs | 105 +++++++++++++++ 3 files changed, 416 insertions(+) create mode 100644 VERIFICATION.md create mode 100644 scripts/verify_ui.mjs diff --git a/.gitignore b/.gitignore index 3e865ad..7616e52 100644 --- a/.gitignore +++ b/.gitignore @@ -106,3 +106,10 @@ site/ # External signature databases (git repositories) signatures/flipperzero-firmware/ signatures/rtl_433/ + +# UI verification screenshots (scripts/verify_ui.mjs) +verify-shots/ + +# Trained model binaries (regenerated by training scripts, not versioned) +models/*.onnx +models/*.pt diff --git a/VERIFICATION.md b/VERIFICATION.md new file mode 100644 index 0000000..6aefcc2 --- /dev/null +++ b/VERIFICATION.md @@ -0,0 +1,304 @@ +# GigLez — Deployable-Artifact Verification + +**Purpose:** prove that the *thing you actually ship* — the Docker container running +`src/api/main_simple.py` — serves the **full user journey**, end to end, the way a real +user experiences it. Unit tests and in-process checks passing is **not** the same as the +shipped app working. This doc is self-contained: you can clear context, come back, and run +it top to bottom. + +Last grounded against the code on **2026-07-20** (branch `master`). + +--- + +## 0. What "the deployable artifact" actually is + +| Layer | What it is | Where | +|-------|-----------|-------| +| Container | `python:3.10-slim`, non-root, torch-free (417 MB) | `Dockerfile` | +| Process | `uvicorn src.api.main_simple:app` on `0.0.0.0:8000` | `docker-compose.yml` | +| Served UI | One SPA page + 6 JS modules | `templates/index.html`, `static/js/{detail-modal,map,upload,search,stats,main}.js` | +| API | FastAPI JSON endpoints (see §3) | `main_simple.py` | +| Data store | JSON file `{captures:[...], upload_counter, last_updated}` on a **named volume** `giglez_data:/app/data` | `data/captures_simple.json` | +| ML model | `models/category_classifier.joblib` (sklearn 1.6.1 / numpy 2.2.6 — pins must match) | baked into image | + +**Runtime external dependencies (important):** the UI pulls Leaflet, Leaflet.markercluster, +Leaflet.heat, and Chart.js from **unpkg/jsdelivr CDNs**, and map tiles from the **OSM tile +server**. The artifact is **not self-contained for offline use** — an air-gapped host will +render a broken map. If offline deploy is ever a requirement, vendor these assets locally. + +--- + +## 1. The full user journey (the 9 things to prove) + +``` +1. BUILD docker compose up --build → container reports healthy +2. LOAD GET / returns the SPA; all 6 JS + CSS assets load; no console errors +3. MAP Leaflet map renders; existing captures show as markers/clusters; stat counters populate +4. UPLOAD user picks a .sub, sets GPS (or "use my location"), submits → progress → result card with a device match +5. IDENTIFY the uploaded capture gets a device_name + device_category + confidence (parse→match→ML ran) +6. PLOT the new capture appears on the map at its GPS +7. SEARCH query by protocol / frequency / geo returns the capture +8. STATS counters + distributions reflect the new data +9. PERSIST restart the container → data survives (named volume did its job) +``` + +Verification is layered so a failure tells you *where* it broke: + +- **§2 Layer 0** — artifact builds & boots +- **§3 Layer 1** — API contract (deterministic curl/jq) → proves the backend serves the journey +- **§4 Layer 2** — browser/UI (Playwright) → proves the *actual UI a user touches* works +- **§5 Layer 3** — persistence across restart +- **§6 Layer 4** — regression guardrails (things fixed recently that must stay fixed) + +--- + +## 2. Layer 0 — Build & boot the real artifact + +```bash +cd ~/coding/security/giglez + +# Build + run exactly what ships +docker compose up --build -d + +# Wait for healthy (compose healthcheck hits /health). Give it ~15s. +docker compose ps # STATUS should show "healthy" +docker inspect --format '{{.State.Health.Status}}' $(docker compose ps -q giglez) +``` + +**PASS:** container status = `healthy`. +**If it fails:** `docker compose logs giglez` — look for import errors (usually a pin +mismatch: the joblib model needs sklearn 1.6.1 / numpy 2.2.6; `requirements-prod.txt`, not +the stale root `requirements.txt`). + +Set a base URL for the rest of the doc: + +```bash +BASE=http://localhost:8000 +``` + +> **Caveat when running inside Claude Code's shell:** the `rtk` hook intercepts *piped* +> `curl` and returns a fake compressed body like `{"ok":true,"top":[]}`. Always write the +> response to a file first (`curl -s "$BASE/..." -o /tmp/r.json`) then inspect with +> `jq . /tmp/r.json`. In a normal human terminal this caveat does not apply. + +--- + +## 3. Layer 1 — API contract checks (deterministic) + +These prove the backend serves stages 2–8 without a browser. Copy-paste each block; the +expected shape is shown. + +### 3.1 Health + API banner + SPA HTML + +```bash +curl -s "$BASE/health" -o /tmp/h.json; jq . /tmp/h.json +# expect: {"status":"healthy","database":"not_connected","mode":"simple"} + +curl -s "$BASE/api" -o /tmp/api.json; jq .status /tmp/api.json +# expect: "operational" + +curl -s "$BASE/" -o /tmp/index.html +grep -c 'id="map"' /tmp/index.html # expect: 1 (SPA shell served) +``` + +### 3.2 Static assets the page actually loads (stage 2) + +```bash +for f in detail-modal map upload search stats main; do + code=$(curl -s -o /dev/null -w '%{http_code}' "$BASE/static/js/$f.js") + echo "$f.js -> $code" # every one must be 200 +done +curl -s -o /dev/null -w 'main.css -> %{http_code}\n' "$BASE/static/css/main.css" +``` + +**PASS:** all `200`. A `404` here = broken page for real users. + +### 3.3 Baseline data snapshot (stages 3 & 8) + +```bash +curl -s "$BASE/api/v1/stats/summary" -o /tmp/s0.json +jq '{total_captures, unique_devices, data_sources}' /tmp/s0.json +BASELINE=$(jq .total_captures /tmp/s0.json); echo "baseline captures = $BASELINE" + +curl -s "$BASE/api/v1/query/captures?limit=1000" -o /tmp/q0.json +jq '.total' /tmp/q0.json # should equal $BASELINE +``` + +### 3.4 Upload → identify → persist (stages 4, 5, 9-write) + +A minimal, self-contained RAW `.sub` (no dependency on the big dataset, which is *excluded* +from the image): + +```bash +cat > /tmp/sample.sub <<'SUB' +Filetype: Flipper SubGhz RAW File +Version: 1 +Frequency: 433920000 +Preset: FuriHalSubGhzPresetOok650Async +Protocol: RAW +RAW_Data: 350 -350 350 -700 700 -350 350 -350 700 -700 350 -350 350 -700 700 -350 350 -700 350 -350 700 -350 350 -700 350 -350 350 -700 700 -700 350 -350 +SUB + +MANIFEST='{"captures":[{"filename":"sample.sub","latitude":47.6062,"longitude":-122.3321,"timestamp":"2026-07-20T12:00:00Z"}]}' + +curl -s -X POST "$BASE/api/v1/captures/upload" \ + -F "files=@/tmp/sample.sub" \ + -F "manifest=$MANIFEST" -o /tmp/up.json + +jq '{success, total_successful, name:.successful[0].device_name, + category:.successful[0].device_category, + conf:.successful[0].match_confidence, + id:.successful[0].id}' /tmp/up.json +``` + +**PASS:** `success=true`, `total_successful=1`, and `device_name` / `device_category` / +`id` are non-null (parse → match → ML category all ran). Note the returned `id`. + +### 3.5 New capture is queryable & plottable (stages 6 & 8) + +```bash +curl -s "$BASE/api/v1/stats/summary" -o /tmp/s1.json +echo "after = $(jq .total_captures /tmp/s1.json) (expect baseline+1 = $((BASELINE+1)))" + +# The capture carries GPS so the map can plot it: +NEWID=$(jq '.successful[0].id' /tmp/up.json) +curl -s "$BASE/api/v1/captures/$NEWID" -o /tmp/det.json +jq '.capture | {id, latitude, longitude, device_category}' /tmp/det.json +``` + +**PASS:** count = baseline+1; the capture has numeric `latitude`/`longitude`. + +### 3.6 Export (bonus) + +```bash +curl -s "$BASE/api/v1/export" -o /tmp/export.ndjson +head -1 /tmp/export.ndjson | jq . # each line is one capture record +``` + +--- + +## 4. Layer 2 — Browser / UI checks (the real user journey) + +The API can be perfect while the UI is broken (bad JS, CDN blocked, map never inits). This +layer drives a **real headless browser** through the DOM. Node Playwright 1.58.2 is present. + +Run the bundled script: + +```bash +cd ~/coding/security/giglez +BASE=http://localhost:8000 node scripts/verify_ui.mjs +``` + +It performs, against real element IDs from `index.html`: + +1. `goto /`, collect **console errors** and **failed network requests**. +2. Assert the Leaflet map container `#map` exists and Leaflet initialised (`.leaflet-container`). +3. Read the `#total-captures` / `#unique-devices` counters. +4. Switch to the Upload section (`a[href="#upload"]`), set `#file-input` to `/tmp/sample.sub`, + fill `#default-lat` / `#default-lon`, click the **Upload** button (`onclick="uploadFiles()"`). +5. Wait for `#upload-results` to become visible and assert it shows a device match. +6. Re-query `/api/v1/query/captures` and assert the total incremented. +7. Save screenshots to `./verify-shots/` at each step. + +**PASS criteria (the script prints a final PASS/FAIL and exits non-zero on failure):** +- 0 console errors, 0 failed asset requests +- `.leaflet-container` present (map actually rendered) +- upload results panel shows a device name + category +- capture total incremented by 1 + +If you'd rather check by eye: open `http://localhost:8000` in a browser, open DevTools +Console (should be clean), confirm the map tiles draw, then walk Upload → Search yourself. + +--- + +## 5. Layer 3 — Persistence across restart (stage 9) + +The whole point of the named volume: a redeploy/restart must not lose user data. + +```bash +BEFORE=$(curl -s "$BASE/api/v1/stats/summary" | jq .total_captures 2>/dev/null); echo "before=$BEFORE" +docker compose restart giglez +sleep 12 +AFTER=$(curl -s "$BASE/api/v1/stats/summary" -o /tmp/sa.json && jq .total_captures /tmp/sa.json) +echo "after=$AFTER" # must equal before +``` + +**PASS:** `after == before`. If it resets, the volume mount or `save_captures()` path is +wrong and every restart wipes contributions. + +> Harder test (proves the volume, not just the file): `docker compose down` (keeps named +> volumes) then `docker compose up -d` and re-check the count. Use `down -v` only when you +> intend to wipe the store. + +--- + +## 6. Layer 4 — Regression guardrails (recently fixed — must stay fixed) + +### 6.1 BinRAW captures produce a category (commit `4396c74`) +BinRAW `.sub` used to yield zero identification. Upload one and confirm a category comes back: + +```bash +cat > /tmp/binraw.sub <<'SUB' +Filetype: Flipper SubGhz RAW File +Version: 1 +Frequency: 433920000 +Preset: FuriHalSubGhzPreset2FSKDev476Async +Protocol: BinRAW +Bit: 48 +TE: 400 +Bit_RAW: 48 +Data_RAW: A9 3C 5F 12 87 D4 +SUB +curl -s -X POST "$BASE/api/v1/captures/upload" \ + -F "files=@/tmp/binraw.sub" \ + -F 'manifest={"captures":[{"filename":"binraw.sub","latitude":47.61,"longitude":-122.33,"timestamp":"2026-07-20T12:05:00Z"}]}' \ + -o /tmp/b.json +jq '.successful[0] | {device_category, match_confidence}' /tmp/b.json +``` +**PASS:** `device_category` is non-null. + +### 6.2 Per-device category shows the device's OWN catalog category (commit `41850b5`) +A named device must not be mislabeled with the ML overall call (e.g. "Acurite … → Fan +Controller"). Inspect the match list from §3.4's upload: + +```bash +jq '.successful[0] | {headline_category: .device_category, + per_device: [.matched_devices[] | {device_name, category}]}' /tmp/up.json +``` +**PASS:** each `matched_devices[].category` is that device's real catalog category; the +top-level `.device_category` may differ (it's the ML overall call) — that's expected. + +--- + +## 7. Pass/fail scorecard + +| # | Stage | Check | Pass condition | +|---|-------|-------|----------------| +| 0 | Build | §2 | container `healthy` | +| 2 | Load | §3.1–3.2 | SPA HTML + all 6 JS + CSS = 200 | +| 3 | Map data | §3.3 | stats + query return baseline | +| 4/5 | Upload+ID | §3.4 | success, device_name+category+id non-null | +| 6/8 | Plot+Stats | §3.5 | count = baseline+1, GPS present | +| 2–8 | **UI** | §4 | Playwright PASS, 0 console errors, map rendered, upload result shown | +| 9 | Persist | §5 | count unchanged after restart | +| R1 | BinRAW | §6.1 | category non-null | +| R2 | Category | §6.2 | per-device catalog category correct | + +All green ⇒ the deployable artifact serves the full user journey and is safe to put in +front of a real user (beta). Remaining true prod gaps are **operational**, not functional: +public host, TLS/reverse proxy, and (optional) vendoring the CDN assets for offline use. + +--- + +## 8. Teardown + +```bash +docker compose down # keep the named volume (data persists) +# docker compose down -v # ONLY if you want to wipe the capture store +rm -f /tmp/sample.sub /tmp/binraw.sub /tmp/*.json /tmp/index.html /tmp/export.ndjson +``` + +**Do not** commit any captures created during verification — the JSON store is baked into +the image as seed data. If you uploaded against a local (non-Docker) `main_simple`, strip +your test records from `data/captures_simple.json` before committing (filter by the test +GPS you used). diff --git a/scripts/verify_ui.mjs b/scripts/verify_ui.mjs new file mode 100644 index 0000000..5094da0 --- /dev/null +++ b/scripts/verify_ui.mjs @@ -0,0 +1,105 @@ +#!/usr/bin/env node +/** + * GigLez UI verification — drives a real headless browser through the full + * user journey against the deployed SPA, asserting the things the API layer + * cannot: the page loads clean, the Leaflet map renders, and an upload done + * *through the actual DOM* returns a device match and increments the store. + * + * Usage: + * BASE=http://localhost:8000 node scripts/verify_ui.mjs + * + * Requires Node Playwright (present: 1.58.x). Exits non-zero on any failure. + * Screenshots are written to ./verify-shots/. + */ +import { chromium } from 'playwright'; +import { writeFileSync, mkdirSync } from 'node:fs'; + +const BASE = process.env.BASE || 'http://localhost:8000'; +const SHOTS = 'verify-shots'; +mkdirSync(SHOTS, { recursive: true }); + +const fails = []; +const ok = (cond, msg) => { console.log(`${cond ? 'PASS' : 'FAIL'} ${msg}`); if (!cond) fails.push(msg); }; + +// A minimal self-contained RAW .sub (no dependency on the big dataset). +const SAMPLE_SUB = [ + 'Filetype: Flipper SubGhz RAW File', + 'Version: 1', + 'Frequency: 433920000', + 'Preset: FuriHalSubGhzPresetOok650Async', + 'Protocol: RAW', + 'RAW_Data: 350 -350 350 -700 700 -350 350 -350 700 -700 350 -350 350 -700 700 -350 350 -700 350 -350 700 -350 350 -700 350 -350 350 -700 700 -700 350 -350', +].join('\n'); + +const consoleErrors = []; +const failedRequests = []; + +const browser = await chromium.launch(); +const ctx = await browser.newContext(); +const page = await ctx.newPage(); + +page.on('console', (m) => { if (m.type() === 'error') consoleErrors.push(m.text()); }); +page.on('requestfailed', (r) => failedRequests.push(`${r.url()} (${r.failure()?.errorText})`)); + +try { + // ---- Stage 2: load ---- + const resp = await page.goto(BASE, { waitUntil: 'networkidle', timeout: 30000 }); + ok(resp && resp.ok(), `GET / returned ${resp && resp.status()}`); + await page.screenshot({ path: `${SHOTS}/1-loaded.png`, fullPage: true }); + + // ---- Stage 3: map rendered ---- + await page.waitForSelector('#map', { timeout: 10000 }); + const leafletUp = await page.locator('#map .leaflet-container').count(); + ok(leafletUp > 0, 'Leaflet map initialised (.leaflet-container present)'); + + const baselineTxt = await page.locator('#total-captures').innerText().catch(() => '0'); + const baseline = parseInt(baselineTxt, 10) || 0; + console.log(` baseline #total-captures = ${baseline}`); + + // ---- Stage 4: switch to Upload, fill the real form ---- + await page.click('a[href="#upload"]').catch(() => {}); + await page.waitForSelector('#upload-section.section.active, #upload-section', { timeout: 5000 }); + + await page.setInputFiles('#file-input', { + name: 'sample.sub', + mimeType: 'application/octet-stream', + buffer: Buffer.from(SAMPLE_SUB), + }); + await page.fill('#default-lat', '47.6062'); + await page.fill('#default-lon', '-122.3321'); + await page.screenshot({ path: `${SHOTS}/2-upload-ready.png`, fullPage: true }); + + // The upload button is a global onclick="uploadFiles()". + await page.click('button[onclick="uploadFiles()"]'); + + // ---- Stage 5: results panel shows a match ---- + await page.waitForSelector('#upload-results', { state: 'visible', timeout: 20000 }); + const resultsTxt = (await page.locator('#upload-results').innerText()).trim(); + ok(resultsTxt.length > 0, 'upload results panel is populated'); + ok(/success|processed|device|match|categ/i.test(resultsTxt), 'results mention a processed/device match'); + await page.screenshot({ path: `${SHOTS}/3-results.png`, fullPage: true }); + + // ---- Stage 6/8: store incremented (checked via the API the UI uses) ---- + const total = await page.evaluate(async () => { + const r = await fetch('/api/v1/query/captures?limit=1000'); + return (await r.json()).total; + }); + ok(total >= baseline + 1, `capture total incremented (${baseline} -> ${total})`); + + // ---- clean-page assertions ---- + ok(consoleErrors.length === 0, `no console errors (saw ${consoleErrors.length})`); + if (consoleErrors.length) consoleErrors.slice(0, 5).forEach((e) => console.log(' console:', e)); + // CDN/tile requests may legitimately fail if offline — report but treat local assets strictly. + const localFailed = failedRequests.filter((u) => u.includes('/static/') || u.includes(BASE)); + ok(localFailed.length === 0, `no failed LOCAL asset requests (saw ${localFailed.length})`); + if (failedRequests.length) failedRequests.slice(0, 8).forEach((u) => console.log(' reqfail:', u)); +} catch (err) { + ok(false, `unhandled error: ${err.message}`); + await page.screenshot({ path: `${SHOTS}/error.png`, fullPage: true }).catch(() => {}); +} finally { + await browser.close(); +} + +writeFileSync(`${SHOTS}/summary.json`, JSON.stringify({ fails, consoleErrors, failedRequests }, null, 2)); +console.log('\n' + (fails.length ? `FAILED (${fails.length}): ${fails.join(' | ')}` : 'ALL CHECKS PASSED')); +process.exit(fails.length ? 1 : 0);