# Rotation cutover proofs This file is the **human-facing mirror** of `internal/rotate/proofs.go`. That Go table is the single source of truth (`incredigo rotate` surfaces it as the **PROOF** column); this document explains what each level *means* and records the basis for every driver's level. Keep the two in sync — if you change one, change the other. ## Why this exists Every `Rotator` in `internal/rotate` contains **only real rotation code**: it talks to a real service (HTTP API, DB client, SSH, local files). None of them contain simulation branches or test awareness — the only thing a test injects is an endpoint / `HTTPClient` / binary path, which a real deployment also configures. What differs between drivers is **how their real-cutover path has been validated**. That is *data, not behaviour*, so it lives in one table (here + `proofs.go`) instead of as prose scattered through — and easily confused with — the driver code. This is the guardrail that stops a mock-only driver from ever being mistaken for a live one. ## Levels | Level | Meaning | |-----------|---------| | **LIVE-VM** | Cutover proven against the **real target software** running in the sandbox VM (real PostgreSQL / MariaDB / Redis / `wg` / Gitea / local files). | | **MOCK-ONLY** | Cutover proven only against an **emulator** — our own `httptest` / in-process SSH server, **or a third-party mock** (e.g. `moto` for AWS). **NOT** proven against the real target service. | | **UNPROVEN** | No cutover proof recorded (e.g. the dry-run noop helper). | > **Honesty note:** running in the VM does **not** automatically make a driver LIVE-VM. > AWS ran in the VM but only against **moto** (a mock AWS), so it is **MOCK-ONLY**. LIVE-VM > means the *real* target software validated the cutover. ## Current status ### LIVE-VM — proven against real target software in the sandbox VM | Driver | Real target | Provisioner | |------------|-------------|-------------| | `postgres` | real PostgreSQL | `lab-provision-pg.sh` | | `mysql` | real MariaDB | `lab-provision-dbclones.sh` | | `redis` | real `redis-server` | `lab-provision-dbclones.sh` | | `wireguard` | real `wg` | `lab-provision-wg.sh` | | `gitea` | real Gitea | `lab-provision-gitea.sh` | | `appsecret` | real local files | `lab-provision-appsec.sh` | | `mongo` | real `mongod` 8.0 | `lab-provision-mongo.sh` | | `k8s` | real k3s `v1.35` kube-apiserver | `lab-provision-k8s.sh` | > **k8s live-POC note (auth-cache timing):** the real kube-apiserver caches *successful* > token authentications for ~10s (`cachedTokenAuthenticator`). After `RevokeOld` deletes the > legacy token Secret, the old token keeps authenticating until that cache entry expires — so > the POC polls the old token for up to 30s and asserts it flips to `401` (it does, ~7s). The > `httptest` emulator invalidates instantly, so this delay only surfaced against real k3s — > exactly what a live POC is for. The driver needed real-CA TLS to talk to the apiserver, so > the blob gained `ca=` (base64 PEM, pinned as the only trusted root) and an `insecure=1` > lab-only escape hatch; neither is test-awareness — a real deployment configures the same CA. ### MOCK-ONLY — proven only against an emulator / mock | Driver | Emulator | Why not (yet) LIVE-VM | |--------------|----------|------------------------| | `aws` | `moto` mock AWS in VM | real AWS never hit | | `sshkey` | in-process SSH server | live VM POC not run | | `openwrt` | in-process SSH server | live QEMU deferred | | `mullvad` | `httptest` emulator | needs a paid account | | `cloudflare` | `httptest` emulator | no self-host | | `ghactions` | `httptest` emulator (holds the libsodium box keypair, decrypts the sealed PUT) | no self-host | | `gitlab` | `httptest` emulator | GitLab CE too heavy for the VM | | `npm` | `httptest` registry emulator | real registry never hit (see caveat below) | | `gcp` | `httptest` IAM emulator (verifies the driver's RS256 JWT against the key's public key) | no self-host | | `twilio` | `httptest` emulator (Basic-auth Keys resource) | no self-host | | `flyio` | `httptest` GraphQL emulator | no self-host | ## What the MOCK-ONLY emulators actually prove The emulators are **not** stubs that rubber-stamp success — each enforces the real protocol so the test exercises the driver's true cutover logic: - **gitlab** — `self/rotate` mints a new PAT *and revokes the caller's old one*; the test asserts `Verify(old)` fails after `Rotate`, proving the revoke really happened. - **cloudflare** — `PUT .../value` rolls the value for the same token id and invalidates the previous value; the test asserts the id is unchanged and `Verify(old)` fails. - **ghactions** — the emulator owns a real NaCl `box` keypair, serves the public key, and on `PUT` **decrypts the sealed value** and stores the plaintext; the test asserts the decrypted value equals the new value the driver put in the rebuilt blob — proving the sealed-box encryption is correct, not simulated. - **npm / twilio** — each emulator enforces credential validity (a token/key authenticates only while it exists) and the test asserts `Verify(old)` fails after `RevokeOld` — proving the create→verify→revoke cutover really happened, old credential dead, new one alive. - **gcp** — the emulator **verifies the RS256 JWT signature** the driver mints against the public key recorded for the key's `kid`; it only issues an access token for a JWT that actually verifies, and `DELETE` removes the key — so the test proves real service-account self-auth (genuine `crypto/rsa` signing) *and* the cutover (revoked key can no longer mint a token). - **flyio** — a managing token mints/deletes the deploy token over GraphQL; the deploy token authenticates the app query only while it exists, so the test proves the managing-token → deploy-token rotation and that the revoked deploy token stops working. > **npm caveat (why it is not just "pending a VM run"):** the public `registry.npmjs.org` > is not self-hostable, and the classic `POST /-/npm/v1/tokens` create-token call on real > npm **requires the account password in the request body** (it is the non-interactive form > of `npm token create`, which prompts for your password). The driver authenticates the > create with the *old token* as Bearer only, which the emulator accepts but real npmjs does > not. So `npm` stays MOCK-ONLY pending either (a) a self-hosted registry that implements the > token API faithfully, or (b) adding password-in-body support to the create call. This is > exactly the kind of contract gap a live-POC attempt is meant to surface. Promoting the rest to LIVE-VM only requires standing up the real service (a self-hosted GitLab CE / a real Cloudflare token / a throwaway GitHub repo) and re-running the same driver against it — no driver code changes. (`k8s` and `mongo` have already made this jump; see the LIVE-VM table above.)