Trilltechnician/giglez
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

test: add end-to-end UI verification harness for deploy artifact

Browse Source 
Adds VERIFICATION.md + scripts/verify_ui.mjs, a Playwright-driven check of
the full user journey against the deployed SPA: page loads clean, Leaflet
map renders, a real DOM upload (.sub + GPS) returns a device match, and the
capture store increments. Grounded in real element IDs and live endpoints.

In-sandbox this session covered the code-correctness layers (L1-substance
parse->match->category, L3 save/load round-trip, L4 BinRAW + per-device
catalog category regressions). The runtime/transport layers still require a
live listening server and must be run from a normal shell:
  - L0            docker compose up --build, container healthy
  - L1 HTTP       curl/jq /health + static assets 200
  - L2            Playwright browser journey (scripts/verify_ui.mjs)
  - L3 restart    container up->down->up, capture count unchanged (volume)

Also ignores trained model binaries (models/*.onnx, *.pt) and the
verify-shots/ screenshot output.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
leetcrypt
2026-07-20 22:30:30 -07:00
parent 41850b5892
commit 58ba021de1
3 changed files with 416 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
VERIFICATION.md
+304
View File
@@ -0,0 +1,304 @@
# GigLez — Deployable-Artifact Verification
**Purpose:** prove that the *thing you actually ship* — the Docker container running
`src/api/main_simple.py` — serves the **full user journey**, end to end, the way a real
user experiences it. Unit tests and in-process checks passing is **not** the same as the
shipped app working. This doc is self-contained: you can clear context, come back, and run
it top to bottom.
Last grounded against the code on **2026-07-20** (branch `master`).
---
## 0. What "the deployable artifact" actually is
| Layer | What it is | Where |
|-------|-----------|-------|
| Container | `python:3.10-slim`, non-root, torch-free (417 MB) | `Dockerfile` |
| Process | `uvicorn src.api.main_simple:app` on `0.0.0.0:8000` | `docker-compose.yml` |
| Served UI | One SPA page + 6 JS modules | `templates/index.html`, `static/js/{detail-modal,map,upload,search,stats,main}.js` |
| API | FastAPI JSON endpoints (see §3) | `main_simple.py` |
| Data store | JSON file `{captures:[...], upload_counter, last_updated}` on a **named volume** `giglez_data:/app/data` | `data/captures_simple.json` |
| ML model | `models/category_classifier.joblib` (sklearn 1.6.1 / numpy 2.2.6 — pins must match) | baked into image |
**Runtime external dependencies (important):** the UI pulls Leaflet, Leaflet.markercluster,
Leaflet.heat, and Chart.js from **unpkg/jsdelivr CDNs**, and map tiles from the **OSM tile
server**. The artifact is **not self-contained for offline use** — an air-gapped host will
render a broken map. If offline deploy is ever a requirement, vendor these assets locally.
---
## 1. The full user journey (the 9 things to prove)
```
1. BUILD docker compose up --build → container reports healthy
2. LOAD GET / returns the SPA; all 6 JS + CSS assets load; no console errors
3. MAP Leaflet map renders; existing captures show as markers/clusters; stat counters populate
4. UPLOAD user picks a .sub, sets GPS (or "use my location"), submits → progress → result card with a device match
5. IDENTIFY the uploaded capture gets a device_name + device_category + confidence (parse→match→ML ran)
6. PLOT the new capture appears on the map at its GPS
7. SEARCH query by protocol / frequency / geo returns the capture
8. STATS counters + distributions reflect the new data
9. PERSIST restart the container → data survives (named volume did its job)
```
Verification is layered so a failure tells you *where* it broke:
- **§2 Layer 0** — artifact builds & boots
- **§3 Layer 1** — API contract (deterministic curl/jq) → proves the backend serves the journey
- **§4 Layer 2** — browser/UI (Playwright) → proves the *actual UI a user touches* works
- **§5 Layer 3** — persistence across restart
- **§6 Layer 4** — regression guardrails (things fixed recently that must stay fixed)
---
## 2. Layer 0 — Build & boot the real artifact
```bash
cd ~/coding/security/giglez
# Build + run exactly what ships
docker compose up --build -d
# Wait for healthy (compose healthcheck hits /health). Give it ~15s.
docker compose ps # STATUS should show "healthy"
docker inspect --format '{{.State.Health.Status}}' $(docker compose ps -q giglez)
```
**PASS:** container status = `healthy`.
**If it fails:** `docker compose logs giglez` — look for import errors (usually a pin
mismatch: the joblib model needs sklearn 1.6.1 / numpy 2.2.6; `requirements-prod.txt`, not
the stale root `requirements.txt`).
Set a base URL for the rest of the doc:
```bash
BASE=http://localhost:8000
```
> **Caveat when running inside Claude Code's shell:** the `rtk` hook intercepts *piped*
> `curl` and returns a fake compressed body like `{"ok":true,"top":[]}`. Always write the
> response to a file first (`curl -s "$BASE/..." -o /tmp/r.json`) then inspect with
> `jq . /tmp/r.json`. In a normal human terminal this caveat does not apply.
---
## 3. Layer 1 — API contract checks (deterministic)
These prove the backend serves stages 2–8 without a browser. Copy-paste each block; the
expected shape is shown.
### 3.1 Health + API banner + SPA HTML
```bash
curl -s "$BASE/health" -o /tmp/h.json; jq . /tmp/h.json
# expect: {"status":"healthy","database":"not_connected","mode":"simple"}
curl -s "$BASE/api" -o /tmp/api.json; jq .status /tmp/api.json
# expect: "operational"
curl -s "$BASE/" -o /tmp/index.html
grep -c 'id="map"' /tmp/index.html # expect: 1 (SPA shell served)
```
### 3.2 Static assets the page actually loads (stage 2)
```bash
for f in detail-modal map upload search stats main; do
code=$(curl -s -o /dev/null -w '%{http_code}' "$BASE/static/js/$f.js")
echo "$f.js -> $code" # every one must be 200
done
curl -s -o /dev/null -w 'main.css -> %{http_code}\n' "$BASE/static/css/main.css"
```
**PASS:** all `200`. A `404` here = broken page for real users.
### 3.3 Baseline data snapshot (stages 3 & 8)
```bash
curl -s "$BASE/api/v1/stats/summary" -o /tmp/s0.json
jq '{total_captures, unique_devices, data_sources}' /tmp/s0.json
BASELINE=$(jq .total_captures /tmp/s0.json); echo "baseline captures = $BASELINE"
curl -s "$BASE/api/v1/query/captures?limit=1000" -o /tmp/q0.json
jq '.total' /tmp/q0.json # should equal $BASELINE
```
### 3.4 Upload → identify → persist (stages 4, 5, 9-write)
A minimal, self-contained RAW `.sub` (no dependency on the big dataset, which is *excluded*
from the image):
```bash
cat > /tmp/sample.sub <<'SUB'
Filetype: Flipper SubGhz RAW File
Version: 1
Frequency: 433920000
Preset: FuriHalSubGhzPresetOok650Async
Protocol: RAW
RAW_Data: 350 -350 350 -700 700 -350 350 -350 700 -700 350 -350 350 -700 700 -350 350 -700 350 -350 700 -350 350 -700 350 -350 350 -700 700 -700 350 -350
SUB
MANIFEST='{"captures":[{"filename":"sample.sub","latitude":47.6062,"longitude":-122.3321,"timestamp":"2026-07-20T12:00:00Z"}]}'
curl -s -X POST "$BASE/api/v1/captures/upload" \
-F "files=@/tmp/sample.sub" \
-F "manifest=$MANIFEST" -o /tmp/up.json
jq '{success, total_successful, name:.successful[0].device_name,
category:.successful[0].device_category,
conf:.successful[0].match_confidence,
id:.successful[0].id}' /tmp/up.json
```
**PASS:** `success=true`, `total_successful=1`, and `device_name` / `device_category` /
`id` are non-null (parse → match → ML category all ran). Note the returned `id`.
### 3.5 New capture is queryable & plottable (stages 6 & 8)
```bash
curl -s "$BASE/api/v1/stats/summary" -o /tmp/s1.json
echo "after = $(jq .total_captures /tmp/s1.json) (expect baseline+1 = $((BASELINE+1)))"
# The capture carries GPS so the map can plot it:
NEWID=$(jq '.successful[0].id' /tmp/up.json)
curl -s "$BASE/api/v1/captures/$NEWID" -o /tmp/det.json
jq '.capture | {id, latitude, longitude, device_category}' /tmp/det.json
```
**PASS:** count = baseline+1; the capture has numeric `latitude`/`longitude`.
### 3.6 Export (bonus)
```bash
curl -s "$BASE/api/v1/export" -o /tmp/export.ndjson
head -1 /tmp/export.ndjson | jq . # each line is one capture record
```
---
## 4. Layer 2 — Browser / UI checks (the real user journey)
The API can be perfect while the UI is broken (bad JS, CDN blocked, map never inits). This
layer drives a **real headless browser** through the DOM. Node Playwright 1.58.2 is present.
Run the bundled script:
```bash
cd ~/coding/security/giglez
BASE=http://localhost:8000 node scripts/verify_ui.mjs
```
It performs, against real element IDs from `index.html`:
1. `goto /`, collect **console errors** and **failed network requests**.
2. Assert the Leaflet map container `#map` exists and Leaflet initialised (`.leaflet-container`).
3. Read the `#total-captures` / `#unique-devices` counters.
4. Switch to the Upload section (`a[href="#upload"]`), set `#file-input` to `/tmp/sample.sub`,
fill `#default-lat` / `#default-lon`, click the **Upload** button (`onclick="uploadFiles()"`).
5. Wait for `#upload-results` to become visible and assert it shows a device match.
6. Re-query `/api/v1/query/captures` and assert the total incremented.
7. Save screenshots to `./verify-shots/` at each step.
**PASS criteria (the script prints a final PASS/FAIL and exits non-zero on failure):**
- 0 console errors, 0 failed asset requests
- `.leaflet-container` present (map actually rendered)
- upload results panel shows a device name + category
- capture total incremented by 1
If you'd rather check by eye: open `http://localhost:8000` in a browser, open DevTools
Console (should be clean), confirm the map tiles draw, then walk Upload → Search yourself.
---
## 5. Layer 3 — Persistence across restart (stage 9)
The whole point of the named volume: a redeploy/restart must not lose user data.
```bash
BEFORE=$(curl -s "$BASE/api/v1/stats/summary" | jq .total_captures 2>/dev/null); echo "before=$BEFORE"
docker compose restart giglez
sleep 12
AFTER=$(curl -s "$BASE/api/v1/stats/summary" -o /tmp/sa.json && jq .total_captures /tmp/sa.json)
echo "after=$AFTER" # must equal before
```
**PASS:** `after == before`. If it resets, the volume mount or `save_captures()` path is
wrong and every restart wipes contributions.
> Harder test (proves the volume, not just the file): `docker compose down` (keeps named
> volumes) then `docker compose up -d` and re-check the count. Use `down -v` only when you
> intend to wipe the store.
---
## 6. Layer 4 — Regression guardrails (recently fixed — must stay fixed)
### 6.1 BinRAW captures produce a category (commit `4396c74`)
BinRAW `.sub` used to yield zero identification. Upload one and confirm a category comes back:
```bash
cat > /tmp/binraw.sub <<'SUB'
Filetype: Flipper SubGhz RAW File
Version: 1
Frequency: 433920000
Preset: FuriHalSubGhzPreset2FSKDev476Async
Protocol: BinRAW
Bit: 48
TE: 400
Bit_RAW: 48
Data_RAW: A9 3C 5F 12 87 D4
SUB
curl -s -X POST "$BASE/api/v1/captures/upload" \
-F "files=@/tmp/binraw.sub" \
-F 'manifest={"captures":[{"filename":"binraw.sub","latitude":47.61,"longitude":-122.33,"timestamp":"2026-07-20T12:05:00Z"}]}' \
-o /tmp/b.json
jq '.successful[0] | {device_category, match_confidence}' /tmp/b.json
```
**PASS:** `device_category` is non-null.
### 6.2 Per-device category shows the device's OWN catalog category (commit `41850b5`)
A named device must not be mislabeled with the ML overall call (e.g. "Acurite … → Fan
Controller"). Inspect the match list from §3.4's upload:
```bash
jq '.successful[0] | {headline_category: .device_category,
per_device: [.matched_devices[] | {device_name, category}]}' /tmp/up.json
```
**PASS:** each `matched_devices[].category` is that device's real catalog category; the
top-level `.device_category` may differ (it's the ML overall call) — that's expected.
---
## 7. Pass/fail scorecard
| # | Stage | Check | Pass condition |
|---|-------|-------|----------------|
| 0 | Build | §2 | container `healthy` |
| 2 | Load | §3.1–3.2 | SPA HTML + all 6 JS + CSS = 200 |
| 3 | Map data | §3.3 | stats + query return baseline |
| 4/5 | Upload+ID | §3.4 | success, device_name+category+id non-null |
| 6/8 | Plot+Stats | §3.5 | count = baseline+1, GPS present |
| 2–8 | **UI** | §4 | Playwright PASS, 0 console errors, map rendered, upload result shown |
| 9 | Persist | §5 | count unchanged after restart |
| R1 | BinRAW | §6.1 | category non-null |
| R2 | Category | §6.2 | per-device catalog category correct |
All green ⇒ the deployable artifact serves the full user journey and is safe to put in
front of a real user (beta). Remaining true prod gaps are **operational**, not functional:
public host, TLS/reverse proxy, and (optional) vendoring the CDN assets for offline use.
---
## 8. Teardown
```bash
docker compose down # keep the named volume (data persists)
# docker compose down -v # ONLY if you want to wipe the capture store
rm -f /tmp/sample.sub /tmp/binraw.sub /tmp/*.json /tmp/index.html /tmp/export.ndjson
```
**Do not** commit any captures created during verification — the JSON store is baked into
the image as seed data. If you uploaded against a local (non-Docker) `main_simple`, strip
your test records from `data/captures_simple.json` before committing (filter by the test
GPS you used).