Trilltechnician/giglez
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

test: add end-to-end UI verification harness for deploy artifact

Browse Source 
Adds VERIFICATION.md + scripts/verify_ui.mjs, a Playwright-driven check of
the full user journey against the deployed SPA: page loads clean, Leaflet
map renders, a real DOM upload (.sub + GPS) returns a device match, and the
capture store increments. Grounded in real element IDs and live endpoints.

In-sandbox this session covered the code-correctness layers (L1-substance
parse->match->category, L3 save/load round-trip, L4 BinRAW + per-device
catalog category regressions). The runtime/transport layers still require a
live listening server and must be run from a normal shell:
  - L0            docker compose up --build, container healthy
  - L1 HTTP       curl/jq /health + static assets 200
  - L2            Playwright browser journey (scripts/verify_ui.mjs)
  - L3 restart    container up->down->up, capture count unchanged (volume)

Also ignores trained model binaries (models/*.onnx, *.pt) and the
verify-shots/ screenshot output.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
leetcrypt
2026-07-20 22:30:30 -07:00
parent 41850b5892
commit 58ba021de1
3 changed files with 416 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitignore
+7
View File
@@ -106,3 +106,10 @@ site/
# External signature databases (git repositories) # External signature databases (git repositories)
signatures/flipperzero-firmware/ signatures/flipperzero-firmware/
signatures/rtl_433/ signatures/rtl_433/
# UI verification screenshots (scripts/verify_ui.mjs)
verify-shots/
# Trained model binaries (regenerated by training scripts, not versioned)
models/*.onnx
models/*.pt
VERIFICATION.md
+304
View File
@@ -0,0 +1,304 @@
# GigLez — Deployable-Artifact Verification
**Purpose:** prove that the *thing you actually ship* — the Docker container running
`src/api/main_simple.py` — serves the **full user journey**, end to end, the way a real
user experiences it. Unit tests and in-process checks passing is **not** the same as the
shipped app working. This doc is self-contained: you can clear context, come back, and run
it top to bottom.
Last grounded against the code on **2026-07-20** (branch `master`).
---
## 0. What "the deployable artifact" actually is
| Layer | What it is | Where |
|-------|-----------|-------|
| Container | `python:3.10-slim`, non-root, torch-free (417 MB) | `Dockerfile` |
| Process | `uvicorn src.api.main_simple:app` on `0.0.0.0:8000` | `docker-compose.yml` |
| Served UI | One SPA page + 6 JS modules | `templates/index.html`, `static/js/{detail-modal,map,upload,search,stats,main}.js` |
| API | FastAPI JSON endpoints (see §3) | `main_simple.py` |
| Data store | JSON file `{captures:[...], upload_counter, last_updated}` on a **named volume** `giglez_data:/app/data` | `data/captures_simple.json` |
| ML model | `models/category_classifier.joblib` (sklearn 1.6.1 / numpy 2.2.6 — pins must match) | baked into image |
**Runtime external dependencies (important):** the UI pulls Leaflet, Leaflet.markercluster,
Leaflet.heat, and Chart.js from **unpkg/jsdelivr CDNs**, and map tiles from the **OSM tile
server**. The artifact is **not self-contained for offline use** — an air-gapped host will
render a broken map. If offline deploy is ever a requirement, vendor these assets locally.
---
## 1. The full user journey (the 9 things to prove)
```
1. BUILD docker compose up --build → container reports healthy
2. LOAD GET / returns the SPA; all 6 JS + CSS assets load; no console errors
3. MAP Leaflet map renders; existing captures show as markers/clusters; stat counters populate
4. UPLOAD user picks a .sub, sets GPS (or "use my location"), submits → progress → result card with a device match
5. IDENTIFY the uploaded capture gets a device_name + device_category + confidence (parse→match→ML ran)
6. PLOT the new capture appears on the map at its GPS
7. SEARCH query by protocol / frequency / geo returns the capture
8. STATS counters + distributions reflect the new data
9. PERSIST restart the container → data survives (named volume did its job)
```
Verification is layered so a failure tells you *where* it broke:
- **§2 Layer 0** — artifact builds & boots
- **§3 Layer 1** — API contract (deterministic curl/jq) → proves the backend serves the journey
- **§4 Layer 2** — browser/UI (Playwright) → proves the *actual UI a user touches* works
- **§5 Layer 3** — persistence across restart
- **§6 Layer 4** — regression guardrails (things fixed recently that must stay fixed)
---
## 2. Layer 0 — Build & boot the real artifact
```bash
cd ~/coding/security/giglez
# Build + run exactly what ships
docker compose up --build -d
# Wait for healthy (compose healthcheck hits /health). Give it ~15s.
docker compose ps # STATUS should show "healthy"
docker inspect --format '{{.State.Health.Status}}' $(docker compose ps -q giglez)
```
**PASS:** container status = `healthy`.
**If it fails:** `docker compose logs giglez` — look for import errors (usually a pin
mismatch: the joblib model needs sklearn 1.6.1 / numpy 2.2.6; `requirements-prod.txt`, not
the stale root `requirements.txt`).
Set a base URL for the rest of the doc:
```bash
BASE=http://localhost:8000
```
> **Caveat when running inside Claude Code's shell:** the `rtk` hook intercepts *piped*
> `curl` and returns a fake compressed body like `{"ok":true,"top":[]}`. Always write the
> response to a file first (`curl -s "$BASE/..." -o /tmp/r.json`) then inspect with
> `jq . /tmp/r.json`. In a normal human terminal this caveat does not apply.
---
## 3. Layer 1 — API contract checks (deterministic)
These prove the backend serves stages 2–8 without a browser. Copy-paste each block; the
expected shape is shown.
### 3.1 Health + API banner + SPA HTML
```bash
curl -s "$BASE/health" -o /tmp/h.json; jq . /tmp/h.json
# expect: {"status":"healthy","database":"not_connected","mode":"simple"}
curl -s "$BASE/api" -o /tmp/api.json; jq .status /tmp/api.json
# expect: "operational"
curl -s "$BASE/" -o /tmp/index.html
grep -c 'id="map"' /tmp/index.html # expect: 1 (SPA shell served)
```
### 3.2 Static assets the page actually loads (stage 2)
```bash
for f in detail-modal map upload search stats main; do
code=$(curl -s -o /dev/null -w '%{http_code}' "$BASE/static/js/$f.js")
echo "$f.js -> $code" # every one must be 200
done
curl -s -o /dev/null -w 'main.css -> %{http_code}\n' "$BASE/static/css/main.css"
```
**PASS:** all `200`. A `404` here = broken page for real users.
### 3.3 Baseline data snapshot (stages 3 & 8)
```bash
curl -s "$BASE/api/v1/stats/summary" -o /tmp/s0.json
jq '{total_captures, unique_devices, data_sources}' /tmp/s0.json
BASELINE=$(jq .total_captures /tmp/s0.json); echo "baseline captures = $BASELINE"
curl -s "$BASE/api/v1/query/captures?limit=1000" -o /tmp/q0.json
jq '.total' /tmp/q0.json # should equal $BASELINE
```
### 3.4 Upload → identify → persist (stages 4, 5, 9-write)
A minimal, self-contained RAW `.sub` (no dependency on the big dataset, which is *excluded*
from the image):
```bash
cat > /tmp/sample.sub <<'SUB'
Filetype: Flipper SubGhz RAW File
Version: 1
Frequency: 433920000
Preset: FuriHalSubGhzPresetOok650Async
Protocol: RAW
RAW_Data: 350 -350 350 -700 700 -350 350 -350 700 -700 350 -350 350 -700 700 -350 350 -700 350 -350 700 -350 350 -700 350 -350 350 -700 700 -700 350 -350
SUB
MANIFEST='{"captures":[{"filename":"sample.sub","latitude":47.6062,"longitude":-122.3321,"timestamp":"2026-07-20T12:00:00Z"}]}'
curl -s -X POST "$BASE/api/v1/captures/upload" \
-F "files=@/tmp/sample.sub" \
-F "manifest=$MANIFEST" -o /tmp/up.json
jq '{success, total_successful, name:.successful[0].device_name,
category:.successful[0].device_category,
conf:.successful[0].match_confidence,
id:.successful[0].id}' /tmp/up.json
```
**PASS:** `success=true`, `total_successful=1`, and `device_name` / `device_category` /
`id` are non-null (parse → match → ML category all ran). Note the returned `id`.
### 3.5 New capture is queryable & plottable (stages 6 & 8)
```bash
curl -s "$BASE/api/v1/stats/summary" -o /tmp/s1.json
echo "after = $(jq .total_captures /tmp/s1.json) (expect baseline+1 = $((BASELINE+1)))"
# The capture carries GPS so the map can plot it:
NEWID=$(jq '.successful[0].id' /tmp/up.json)
curl -s "$BASE/api/v1/captures/$NEWID" -o /tmp/det.json
jq '.capture | {id, latitude, longitude, device_category}' /tmp/det.json
```
**PASS:** count = baseline+1; the capture has numeric `latitude`/`longitude`.
### 3.6 Export (bonus)
```bash
curl -s "$BASE/api/v1/export" -o /tmp/export.ndjson
head -1 /tmp/export.ndjson | jq . # each line is one capture record
```
---
## 4. Layer 2 — Browser / UI checks (the real user journey)
The API can be perfect while the UI is broken (bad JS, CDN blocked, map never inits). This
layer drives a **real headless browser** through the DOM. Node Playwright 1.58.2 is present.
Run the bundled script:
```bash
cd ~/coding/security/giglez
BASE=http://localhost:8000 node scripts/verify_ui.mjs
```
It performs, against real element IDs from `index.html`:
1. `goto /`, collect **console errors** and **failed network requests**.
2. Assert the Leaflet map container `#map` exists and Leaflet initialised (`.leaflet-container`).
3. Read the `#total-captures` / `#unique-devices` counters.
4. Switch to the Upload section (`a[href="#upload"]`), set `#file-input` to `/tmp/sample.sub`,
fill `#default-lat` / `#default-lon`, click the **Upload** button (`onclick="uploadFiles()"`).
5. Wait for `#upload-results` to become visible and assert it shows a device match.
6. Re-query `/api/v1/query/captures` and assert the total incremented.
7. Save screenshots to `./verify-shots/` at each step.
**PASS criteria (the script prints a final PASS/FAIL and exits non-zero on failure):**
- 0 console errors, 0 failed asset requests
- `.leaflet-container` present (map actually rendered)
- upload results panel shows a device name + category
- capture total incremented by 1
If you'd rather check by eye: open `http://localhost:8000` in a browser, open DevTools
Console (should be clean), confirm the map tiles draw, then walk Upload → Search yourself.
---
## 5. Layer 3 — Persistence across restart (stage 9)
The whole point of the named volume: a redeploy/restart must not lose user data.
```bash
BEFORE=$(curl -s "$BASE/api/v1/stats/summary" | jq .total_captures 2>/dev/null); echo "before=$BEFORE"
docker compose restart giglez
sleep 12
AFTER=$(curl -s "$BASE/api/v1/stats/summary" -o /tmp/sa.json && jq .total_captures /tmp/sa.json)
echo "after=$AFTER" # must equal before
```
**PASS:** `after == before`. If it resets, the volume mount or `save_captures()` path is
wrong and every restart wipes contributions.
> Harder test (proves the volume, not just the file): `docker compose down` (keeps named
> volumes) then `docker compose up -d` and re-check the count. Use `down -v` only when you
> intend to wipe the store.
---
## 6. Layer 4 — Regression guardrails (recently fixed — must stay fixed)
### 6.1 BinRAW captures produce a category (commit `4396c74`)
BinRAW `.sub` used to yield zero identification. Upload one and confirm a category comes back:
```bash
cat > /tmp/binraw.sub <<'SUB'
Filetype: Flipper SubGhz RAW File
Version: 1
Frequency: 433920000
Preset: FuriHalSubGhzPreset2FSKDev476Async
Protocol: BinRAW
Bit: 48
TE: 400
Bit_RAW: 48
Data_RAW: A9 3C 5F 12 87 D4
SUB
curl -s -X POST "$BASE/api/v1/captures/upload" \
-F "files=@/tmp/binraw.sub" \
-F 'manifest={"captures":[{"filename":"binraw.sub","latitude":47.61,"longitude":-122.33,"timestamp":"2026-07-20T12:05:00Z"}]}' \
-o /tmp/b.json
jq '.successful[0] | {device_category, match_confidence}' /tmp/b.json
```
**PASS:** `device_category` is non-null.
### 6.2 Per-device category shows the device's OWN catalog category (commit `41850b5`)
A named device must not be mislabeled with the ML overall call (e.g. "Acurite … → Fan
Controller"). Inspect the match list from §3.4's upload:
```bash
jq '.successful[0] | {headline_category: .device_category,
per_device: [.matched_devices[] | {device_name, category}]}' /tmp/up.json
```
**PASS:** each `matched_devices[].category` is that device's real catalog category; the
top-level `.device_category` may differ (it's the ML overall call) — that's expected.
---
## 7. Pass/fail scorecard
| # | Stage | Check | Pass condition |
|---|-------|-------|----------------|
| 0 | Build | §2 | container `healthy` |
| 2 | Load | §3.1–3.2 | SPA HTML + all 6 JS + CSS = 200 |
| 3 | Map data | §3.3 | stats + query return baseline |
| 4/5 | Upload+ID | §3.4 | success, device_name+category+id non-null |
| 6/8 | Plot+Stats | §3.5 | count = baseline+1, GPS present |
| 2–8 | **UI** | §4 | Playwright PASS, 0 console errors, map rendered, upload result shown |
| 9 | Persist | §5 | count unchanged after restart |
| R1 | BinRAW | §6.1 | category non-null |
| R2 | Category | §6.2 | per-device catalog category correct |
All green ⇒ the deployable artifact serves the full user journey and is safe to put in
front of a real user (beta). Remaining true prod gaps are **operational**, not functional:
public host, TLS/reverse proxy, and (optional) vendoring the CDN assets for offline use.
---
## 8. Teardown
```bash
docker compose down # keep the named volume (data persists)
# docker compose down -v # ONLY if you want to wipe the capture store
rm -f /tmp/sample.sub /tmp/binraw.sub /tmp/*.json /tmp/index.html /tmp/export.ndjson
```
**Do not** commit any captures created during verification — the JSON store is baked into
the image as seed data. If you uploaded against a local (non-Docker) `main_simple`, strip
your test records from `data/captures_simple.json` before committing (filter by the test
GPS you used).
scripts/verify_ui.mjs
+105
View File
@@ -0,0 +1,105 @@
#!/usr/bin/env node
/**
* GigLez UI verification — drives a real headless browser through the full
* user journey against the deployed SPA, asserting the things the API layer
* cannot: the page loads clean, the Leaflet map renders, and an upload done
* *through the actual DOM* returns a device match and increments the store.
*
* Usage:
* BASE=http://localhost:8000 node scripts/verify_ui.mjs
*
* Requires Node Playwright (present: 1.58.x). Exits non-zero on any failure.
* Screenshots are written to ./verify-shots/.
*/
import { chromium } from 'playwright';
import { writeFileSync, mkdirSync } from 'node:fs';
const BASE = process.env.BASE || 'http://localhost:8000';
const SHOTS = 'verify-shots';
mkdirSync(SHOTS, { recursive: true });
const fails = [];
const ok = (cond, msg) => { console.log(`${cond ? 'PASS' : 'FAIL'} ${msg}`); if (!cond) fails.push(msg); };
// A minimal self-contained RAW .sub (no dependency on the big dataset).
const SAMPLE_SUB = [
'Filetype: Flipper SubGhz RAW File',
'Version: 1',
'Frequency: 433920000',
'Preset: FuriHalSubGhzPresetOok650Async',
'Protocol: RAW',
'RAW_Data: 350 -350 350 -700 700 -350 350 -350 700 -700 350 -350 350 -700 700 -350 350 -700 350 -350 700 -350 350 -700 350 -350 350 -700 700 -700 350 -350',
].join('\n');
const consoleErrors = [];
const failedRequests = [];
const browser = await chromium.launch();
const ctx = await browser.newContext();
const page = await ctx.newPage();
page.on('console', (m) => { if (m.type() === 'error') consoleErrors.push(m.text()); });
page.on('requestfailed', (r) => failedRequests.push(`${r.url()} (${r.failure()?.errorText})`));
try {
// ---- Stage 2: load ----
const resp = await page.goto(BASE, { waitUntil: 'networkidle', timeout: 30000 });
ok(resp && resp.ok(), `GET / returned ${resp && resp.status()}`);
await page.screenshot({ path: `${SHOTS}/1-loaded.png`, fullPage: true });
// ---- Stage 3: map rendered ----
await page.waitForSelector('#map', { timeout: 10000 });
const leafletUp = await page.locator('#map .leaflet-container').count();
ok(leafletUp > 0, 'Leaflet map initialised (.leaflet-container present)');
const baselineTxt = await page.locator('#total-captures').innerText().catch(() => '0');
const baseline = parseInt(baselineTxt, 10) || 0;
console.log(` baseline #total-captures = ${baseline}`);
// ---- Stage 4: switch to Upload, fill the real form ----
await page.click('a[href="#upload"]').catch(() => {});
await page.waitForSelector('#upload-section.section.active, #upload-section', { timeout: 5000 });
await page.setInputFiles('#file-input', {
name: 'sample.sub',
mimeType: 'application/octet-stream',
buffer: Buffer.from(SAMPLE_SUB),
});
await page.fill('#default-lat', '47.6062');
await page.fill('#default-lon', '-122.3321');
await page.screenshot({ path: `${SHOTS}/2-upload-ready.png`, fullPage: true });
// The upload button is a global onclick="uploadFiles()".
await page.click('button[onclick="uploadFiles()"]');
// ---- Stage 5: results panel shows a match ----
await page.waitForSelector('#upload-results', { state: 'visible', timeout: 20000 });
const resultsTxt = (await page.locator('#upload-results').innerText()).trim();
ok(resultsTxt.length > 0, 'upload results panel is populated');
ok(/success|processed|device|match|categ/i.test(resultsTxt), 'results mention a processed/device match');
await page.screenshot({ path: `${SHOTS}/3-results.png`, fullPage: true });
// ---- Stage 6/8: store incremented (checked via the API the UI uses) ----
const total = await page.evaluate(async () => {
const r = await fetch('/api/v1/query/captures?limit=1000');
return (await r.json()).total;
});
ok(total >= baseline + 1, `capture total incremented (${baseline} -> ${total})`);
// ---- clean-page assertions ----
ok(consoleErrors.length === 0, `no console errors (saw ${consoleErrors.length})`);
if (consoleErrors.length) consoleErrors.slice(0, 5).forEach((e) => console.log(' console:', e));
// CDN/tile requests may legitimately fail if offline — report but treat local assets strictly.
const localFailed = failedRequests.filter((u) => u.includes('/static/') || u.includes(BASE));
ok(localFailed.length === 0, `no failed LOCAL asset requests (saw ${localFailed.length})`);
if (failedRequests.length) failedRequests.slice(0, 8).forEach((u) => console.log(' reqfail:', u));
} catch (err) {
ok(false, `unhandled error: ${err.message}`);
await page.screenshot({ path: `${SHOTS}/error.png`, fullPage: true }).catch(() => {});
} finally {
await browser.close();
}
writeFileSync(`${SHOTS}/summary.json`, JSON.stringify({ fails, consoleErrors, failedRequests }, null, 2));
console.log('\n' + (fails.length ? `FAILED (${fails.length}): ${fails.join(' | ')}` : 'ALL CHECKS PASSED'));
process.exit(fails.length ? 1 : 0);