incredigo/docs/ROTATION-PROOFS.md

# Rotation cutover proofs

This file is the **human-facing mirror** of `internal/rotate/proofs.go`. That Go table is
the single source of truth (`incredigo rotate` surfaces it as the **PROOF** column); this
document explains what each level *means* and records the basis for every driver's level.
Keep the two in sync — if you change one, change the other.

## Why this exists

Every `Rotator` in `internal/rotate` contains **only real rotation code**: it talks to a
real service (HTTP API, DB client, SSH, local files). None of them contain simulation
branches or test awareness — the only thing a test injects is an endpoint / `HTTPClient` /
binary path, which a real deployment also configures.

What differs between drivers is **how their real-cutover path has been validated**. That is
*data, not behaviour*, so it lives in one table (here + `proofs.go`) instead of as prose
scattered through — and easily confused with — the driver code. This is the guardrail that
stops a mock-only driver from ever being mistaken for a live one.

## Levels

| Level | Meaning |
|-----------|---------|
| **LIVE-VM** | Cutover proven against the **real target software** running in the sandbox VM (real PostgreSQL / MariaDB / Redis / `wg` / Gitea / local files). |
| **MOCK-ONLY** | Cutover proven only against an **emulator** — our own `httptest` / in-process SSH server, **or a third-party mock** (e.g. `moto` for AWS). **NOT** proven against the real target service. |
| **UNPROVEN** | No cutover proof recorded (e.g. the dry-run noop helper). |

> **Honesty note:** running in the VM does **not** automatically make a driver LIVE-VM.
> AWS ran in the VM but only against **moto** (a mock AWS), so it is **MOCK-ONLY**. LIVE-VM
> means the *real* target software validated the cutover.

## Current status

### LIVE-VM — proven against real target software in the sandbox VM

| Driver | Real target | Provisioner |
|------------|-------------|-------------|
| `postgres` | real PostgreSQL | `lab-provision-pg.sh` |
| `mysql` | real MariaDB | `lab-provision-dbclones.sh` |
| `redis` | real `redis-server` | `lab-provision-dbclones.sh` |
| `wireguard` | real `wg` | `lab-provision-wg.sh` |
| `gitea` | real Gitea | `lab-provision-gitea.sh` |
| `appsecret` | real local files | `lab-provision-appsec.sh` |
| `mongo` | real `mongod` 8.0 | `lab-provision-mongo.sh` |
| `k8s` | real k3s `v1.35` kube-apiserver | `lab-provision-k8s.sh` |

> **k8s live-POC note (auth-cache timing):** the real kube-apiserver caches *successful*
> token authentications for ~10s (`cachedTokenAuthenticator`). After `RevokeOld` deletes the
> legacy token Secret, the old token keeps authenticating until that cache entry expires — so
> the POC polls the old token for up to 30s and asserts it flips to `401` (it does, ~7s). The
> `httptest` emulator invalidates instantly, so this delay only surfaced against real k3s —
> exactly what a live POC is for. The driver needed real-CA TLS to talk to the apiserver, so
> the blob gained `ca=` (base64 PEM, pinned as the only trusted root) and an `insecure=1`
> lab-only escape hatch; neither is test-awareness — a real deployment configures the same CA.

### MOCK-ONLY — proven only against an emulator / mock

| Driver | Emulator | Why not (yet) LIVE-VM |
|--------------|----------|------------------------|
| `aws` | `moto` mock AWS in VM | real AWS never hit |
| `sshkey` | in-process SSH server | live VM POC not run |
| `openwrt` | in-process SSH server | live QEMU deferred |
| `mullvad` | `httptest` emulator | needs a paid account |
| `cloudflare` | `httptest` emulator | no self-host |
| `ghactions` | `httptest` emulator (holds the libsodium box keypair, decrypts the sealed PUT) | no self-host |
| `gitlab` | `httptest` emulator | GitLab CE too heavy for the VM |
| `npm` | `httptest` registry emulator | real registry never hit (see caveat below) |
| `gcp` | `httptest` IAM emulator (verifies the driver's RS256 JWT against the key's public key) | no self-host |
| `twilio` | `httptest` emulator (Basic-auth Keys resource) | no self-host |
| `flyio` | `httptest` GraphQL emulator | no self-host |
| `resend` | `httptest` emulator (Bearer `/api-keys` resource) | no self-host |

## What the MOCK-ONLY emulators actually prove

The emulators are **not** stubs that rubber-stamp success — each enforces the real
protocol so the test exercises the driver's true cutover logic:

- **gitlab** — `self/rotate` mints a new PAT *and revokes the caller's old one*; the test
  asserts `Verify(old)` fails after `Rotate`, proving the revoke really happened.
- **cloudflare** — `PUT .../value` rolls the value for the same token id and invalidates
  the previous value; the test asserts the id is unchanged and `Verify(old)` fails.
- **ghactions** — the emulator owns a real NaCl `box` keypair, serves the public key, and
  on `PUT` **decrypts the sealed value** and stores the plaintext; the test asserts the
  decrypted value equals the new value the driver put in the rebuilt blob — proving the
  sealed-box encryption is correct, not simulated.
- **npm / twilio / resend** — each emulator enforces credential validity (a token/key
  authenticates only while it exists) and the test asserts `Verify(old)` fails after
  `RevokeOld` — proving the create→verify→revoke cutover really happened, old credential
  dead, new one alive. (Resend additionally proves `Verify(new)` by listing keys with the
  new Bearer token and asserting the new id is present — auth success *and* the key exists.)
- **gcp** — the emulator **verifies the RS256 JWT signature** the driver mints against the
  public key recorded for the key's `kid`; it only issues an access token for a JWT that
  actually verifies, and `DELETE` removes the key — so the test proves real service-account
  self-auth (genuine `crypto/rsa` signing) *and* the cutover (revoked key can no longer mint
  a token).
- **flyio** — a managing token mints/deletes the deploy token over GraphQL; the deploy token
  authenticates the app query only while it exists, so the test proves the managing-token →
  deploy-token rotation and that the revoked deploy token stops working.

> **npm caveat (why it is not just "pending a VM run"):** the public `registry.npmjs.org`
> is not self-hostable, and the classic `POST /-/npm/v1/tokens` create-token call on real
> npm **requires the account password in the request body** (it is the non-interactive form
> of `npm token create`, which prompts for your password). The driver authenticates the
> create with the *old token* as Bearer only, which the emulator accepts but real npmjs does
> not. So `npm` stays MOCK-ONLY pending either (a) a self-hosted registry that implements the
> token API faithfully, or (b) adding password-in-body support to the create call. This is
> exactly the kind of contract gap a live-POC attempt is meant to surface.

Promoting the rest to LIVE-VM only requires standing up the real service (a self-hosted
GitLab CE / a real Cloudflare token / a throwaway GitHub repo) and re-running the same
driver against it — no driver code changes. (`k8s` and `mongo` have already made this jump;
see the LIVE-VM table above.)