# Findings — controlled-environment test run (2026-06-16) Source: `~/incredigo-lab-stage/lab-harness.sh` (fresh isolated gopass store + fake, canary-tagged inputs for all 8 scanners). FAKE creds only; never touches a real store. ## Live rotation POC — first real driver (postgres), 2026-06-16 **Result: real end-to-end cutover proven** in the `incredigo-sbx` Multipass VM (real gopasspw/gopass 1.15.16, real PostgreSQL 16, isolated `GOPASS_HOMEDIR`/`GNUPGHOME`, FAKE `labapp@labdb` role). Flow exercised by the shipped binary via `incredigo rotate --execute` (gated: `INCREDIGO_ALLOW_EXECUTE=1` + `--execute`): 1. **Backup gate** sealed + verified 1 entry → `~/.incredigo/rotation-backups/.age` before any change (Hard Rule #1). 2. **Rotate** ran `ALTER USER labapp WITH PASSWORD '<48-hex>'` over the OLD connection. 3. **Verify(new)** → `SELECT 1` with the new secret passed. 4. **Store** the rebuilt DSN back into gopass at the same entry; **re-read + Verify(stored)** passed. 5. **RevokeOld** = no-op (in-place ALTER is itself the cutover). Cutover assertions (all passed): - OLD password (`oldpw835`) **rejected** after rotation — `FATAL: password authentication failed`. - NEW 24-byte hex password (read back from gopass) **authenticates**. - Audit log (`rotate-execute`) contains **no password substring** (old or new). Spine: `internal/rotate/execute.go` (`Execute`), driver `internal/rotate/postgres.go` (self-registers via `init`), sourcing `cmd/incredigo` `gopassCredsForExecute` (Mode A — reconstructs `Source` from the gopass path segment), persistence `sink.Gopass.Show`/`InsertAt`. Unit proof of the same cutover (no live DB): `internal/rotate/postgres_test.go:TestPostgresRotateRealCutover`. Full suite: 45 green. ## Live rotation POC — gitea PAT driver (provider-API pattern), 2026-06-17 **Result: real create-new → verify → revoke-old cutover proven** against a real Gitea 1.25.0 (sqlite, http on loopback) inside the `incredigo-sbx` VM, isolated gopass store, self-owned `labadmin` account. This is the **first provider-API driver** and the first whose `RevokeOld` is a real destructive API call. Provision: `~/incredigo-lab-stage/lab-provision-gitea.sh` (stands up Gitea, mints a seed token via basic auth, seeds the driver-ready blob into gopass). Flow exercised by the shipped binary via `incredigo rotate --execute` (gated): 1. **Backup gate** sealed + verified 1 entry before any change (Hard Rule #1). 2. **Rotate** `POST /api/v1/users/labadmin/tokens` (basic auth) → new token. 3. **Verify(new)** `GET /api/v1/user` with the new token → 200, `login=labadmin`. 4. **Store** the rebuilt blob back into gopass; **re-read + Verify(stored)** passed. 5. **RevokeOld** `DELETE …/tokens/` — a real token deletion. Cutover assertions (all passed): - OLD token → `GET /user` returns **401** after rotation (revoked). - NEW token (read back from gopass) → **200**, `login=labadmin`. - Gitea token list now shows only `incredigo-rotated-`; the seed token is gone. - Audit log (`rotate-execute`) contains **no token substring** (old or new); old ≠ new. Spine: `internal/rotate/gitea.go` (single-line `https://user:token@host/?name=&pw=` blob so it round-trips the existing single-line gopass sink; `authManage` prefers basic auth when a mgmt password is present, else token auth). Discovery for the gitea PAT type: `internal/discover/tea.go` (Gitea `tea` CLI config → driver-ready blob, Source `gitea`). Unit proof of the same enforced cutover (httptest Gitea emulator that 401s a revoked token): `internal/rotate/gitea_test.go`. ## SSH key driver (unit-proven cutover), 2026-06-17 `internal/rotate/sshkey.go` mints a new ed25519 keypair **entirely in memory** (golang.org/x/crypto/ssh; private key never touches disk — Hard Rule #3), installs only the new PUBLIC key into `authorized_keys`, verifies by SSH-dialing the authority, and `RevokeOld` removes the old public key (atomic rewrite). Proven by an **in-process SSH server** test: old key rejected after revoke, new key works, no `PRIVATE KEY` material in `authorized_keys` (`internal/rotate/sshkey_test.go`). Discovery broadened (`internal/discover/ssh.go`): content-based key detection (not just `id_*`) + `~/.ssh/config` `IdentityFile` parsing. Live VM POC not yet run. Full suite after all three drivers: **50 green**, `-race` clean on rotate/sink/vault/discover. ## Live rotation POC — AWS IAM access key driver (cloud-key pattern), 2026-06-17 **Result: real create-new → verify → revoke-old cutover proven** against a self-hosted **moto 5.2.2** mock AWS (`moto_server` on http://127.0.0.1:5000) inside the `incredigo-sbx` VM, isolated gopass store, self-created IAM user `labapp`. This is the cleanest provider-API driver: the OLD credential **self-identifies** — an access key's `AccessKeyId` is non-secret and rides in the secret blob, so RevokeOld deletes exactly the right key (no name-capture dance Gitea needed). Implementation note: the driver speaks the AWS **Query protocol** (form-POST `Action=…&Version=…`) signed with **hand-rolled SigV4** (`crypto/hmac`) — zero new go.mod deps, keeping the static binary lean (the AWS SDK v2 would have added ~50 indirect deps). Spine: `internal/rotate/aws.go`. Secret blob is the single line `aws://:@aws/?region=[&endpoint=]` so it round-trips the single-line gopass sink; `endpoint=` overrides the API base for moto/LocalStack, absent → real `iam/sts.amazonaws.com`. Discovery broadened: `internal/discover/aws.go` now emits that driver-ready blob (was the bare secret) and resolves region from env / `~/.aws/config`. Flow exercised by the shipped binary via `incredigo rotate --execute` (gated): 1. **Backup gate** sealed + verified 1 entry before any change (Hard Rule #1). 2. **Rotate** IAM `CreateAccessKey` (no UserName → moto resolves the calling identity from the signing key) → new key pair. 3. **Verify(new)** STS `GetCallerIdentity` with the new key → ARN returned. 4. **Store** the rebuilt blob into gopass; **re-read + Verify(stored)** passed. 5. **RevokeOld** IAM `DeleteAccessKey` targeting the OLD `AccessKeyId` — a real delete. Cutover assertions (all passed, `lab-verify-aws.sh`): - After rotate, IAM `ListAccessKeys(labapp)` shows **only** the new key; the old `AccessKeyId` is **gone** at the provider. old ≠ new. - New key (read back from gopass) authenticates; audit log (`rotate-execute`) carries **no AccessKeyId and no SecretAccessKey**. **moto enforcement caveat (investigated, important).** moto's STS `GetCallerIdentity` does **not** validate access-key existence — a *deleted* key still returns a (generic `user/moto`) ARN rather than a 403. So against moto, the driver's STS-based Verify cannot itself prove "old key dead"; the live cutover proof is therefore **IAM `ListAccessKeys`** (ground truth that the delete took effect), asserted in `lab-verify-aws.sh`. The *enforced* 403-on-revoked-key path (what real AWS does) is proven instead by the **unit test**: `internal/rotate/aws_test.go`'s `awsEmu` reads the `AccessKeyId` from the SigV4 `Credential=` header and rejects any key not currently present, so `TestAWSRotateRealCutover` asserts the old key returns 403 after revoke, the new key still authenticates, and old ≠ new. The driver's Verify (plain "ARN non-empty") is correct for real AWS, which **does** 403 an invalid key; no moto-specific leniency was baked into the driver. Provision: `~/incredigo-lab-stage/lab-provision-aws.sh` (starts moto, creates `labapp` + seed key via boto3, seeds the driver-ready blob into isolated gopass); verify: `~/incredigo-lab-stage/lab-verify-aws.sh`. Unit proofs: `TestAWSRotateRealCutover` (enforced cutover), `TestAWSSecretRoundTrip` (blob carries `/`+`=` secrets intact), `TestSigV4HeadersStable` (Authorization shape). Full suite after four drivers: **53 green**, `-race` clean on rotate/sink/vault/discover. ## VPN / router / DB-clone batch — 5 drivers, 3 live cutovers, 2026-06-18 Approved expansion (VPN + router secondary persona, plus Tier-1 DB clones). Five new one-file drivers, each with a table-driven unit test (cutover proof + leak-check). Three were additionally proven **live** in the `incredigo-sbx` VM against real services; one is unit-proven with a live QEMU POC deferred; one is unit-only by design. | Driver | Pattern | Live POC | Proof | |--------|---------|----------|-------| | `mysql` | in-place DB password | **PASS** (real MariaDB) | `lab-provision-dbclones.sh` | | `redis` | in-place `CONFIG SET requirepass` | **PASS** (real redis-server) | `lab-provision-dbclones.sh` | | `wireguard` | local keypair regen + peer propagate | **PASS** (real `wg`) | `lab-provision-wg.sh` | | `openwrt` | in-place SSH `passwd` | deferred (QEMU on 1-CPU VM) | unit: in-process SSH server | | `mullvad` | provider-API device key | n/a (needs paid acct) | unit: httptest TLS emulator | **MySQL + Redis live cutover (`lab-provision-dbclones.sh`, all assertions PASS).** Real `mariadb-server` + `redis-server`, isolated gopass, fake `labapp@labdb` and a fake redis `requirepass`. Backup gate sealed+verified first; `rotate --execute` flipped both; the OLD password is **rejected** and the NEW one (read back via `gopass show -o`) **authenticates** for each engine. > **Driver gap found & fixed — MariaDB unprivileged self-rotation.** MariaDB rejects > `ALTER USER CURRENT_USER() IDENTIFIED BY …` for a non-admin (ERROR 1227 — it demands the > global `CREATE USER` privilege even for one's *own* account), which is exactly the > target persona's situation (a vibe coder's DB user is scoped to one schema with > `GRANT ALL ON app.*`). The MySQL-only `USER()` spelling also fails on MariaDB (ERROR > 1064). Fix (`internal/rotate/mysql.go`): try a fallback chain — `ALTER USER > CURRENT_USER()` (MySQL self-service) → `SET PASSWORD = PASSWORD('…')` (MariaDB / MySQL > 5.7) → `SET PASSWORD = '…'` (MySQL 8 / MariaDB ≥10.4); first to succeed wins. After the > fix the live POC passed on MariaDB. **WireGuard live cutover (`lab-provision-wg.sh`, all assertions PASS).** Real `wireguard-tools`: `wg0` = ours, `wg1` = the peer that trusts us. Seeds `wg0`'s old private key into isolated gopass, `rotate --execute` mints a new clamped curve25519 keypair, applies the new private key to `wg0` (via stdin → `wg set wg0 private-key /dev/stdin`, never argv), and re-points the peer to the new public key (carrying over allowed-ips). Assertions: private key changed; peer trusts NEW pubkey; peer **dropped** OLD pubkey; and the headline compatibility proof — **`wg0`'s live interface public key equals the public key our Go code derived**, i.e. our in-process curve25519 derivation matches real `wg pubkey`. > **POC scripting gotcha (Mode A prefix).** Mode A reconstructs the driver `Source` from > the **first** path segment after `--prefix` (`imported//`). The WG POC > first ran with `--prefix imported/wireguard/`, which made `Source="wg0"` (no driver > Detected it → ROTATED 0). Fix: point `--prefix` at the `imported/` **root** so > `imported/wireguard/wg0` resolves `Source="wireguard"`. This is a harness fix, not a > driver bug, but worth recording: narrowing the prefix below the source dir breaks > source recovery. **OpenWrt (unit-proven, live deferred).** `internal/rotate/openwrt.go` dials the router with the OLD password and runs BusyBox `passwd`, feeding the new password twice on the SSH-encrypted stdin (never argv); setting the password IS the cutover (RevokeOld no-op). Proven by an **in-process SSH server** (`openwrt_test.go`) with a mutable current password: old password stops authenticating after rotate, new works, no secret in argv or error strings. A live OpenWrt-in-QEMU POC is the heaviest (full-system emulation on the 1-CPU sandbox VM) and is **deferred**. **Mullvad (unit-only by design).** `internal/rotate/mullvad.go` is the provider-API device-key pattern: fetch an access token from the account number, `POST /accounts/v1/devices {pubkey}` to register a freshly-derived WireGuard key, verify the device lists that exact pubkey, then `DELETE` the old device. Proven against an httptest **TLS** emulator (`mullvad_test.go`) that inspects every request body for the private key (never sent) and 404s a deleted device. No live POC — that needs a real paid Mullvad account; the API contract is faithfully emulated instead. **Discovery broadened (Hard-rule standing directive — dynamic discovery per type).** The env scanner (`internal/discover/env.go`) now tags DB connection URLs by scheme: `postgres`/`postgresql` → `postgres`, `mysql`/`mariadb` → `mysql`, `redis`/`rediss` → `redis` (requires a `user:password` authority), so `DATABASE_URL`/`REDIS_URL`/`MYSQL_URL` in a `.env` route straight to the matching in-place driver, kept regardless of the secret-name heuristic. Regression: `scanners_test.go:TestEnvScanner`. Full suite after the batch: **68 green**, `-race` clean on rotate/sink/vault/discover. ## SaaS-token / app-signing batch — 4 drivers + proof manifest, 2026-06-18 Four more drivers, each one-file + table-driven test (cutover proof + leak-check). User direction this batch: **keep real-rotation code unmistakably distinct from mock code, and record the real-vs-mock distinction as DATA, not behaviour.** Implemented as a single Go table (`internal/rotate/proofs.go`) + a human mirror (`docs/ROTATION-PROOFS.md`), surfaced as a **PROOF** column in `incredigo rotate` so a mock-only driver can never be mistaken for a live one. - **gitlab** (`gitlab.go`, MOCK-ONLY) — GitLab PAT via `POST /api/v4/personal_access_tokens/self/rotate` (`PRIVATE-TOKEN` header). GitLab mints the new token *and revokes the caller's old one in the same call* → in-place, `RevokeOld` no-op. Blob `gitlab:///?tok=`. The `httptest` emulator enforces the revoke, so the unit test's `Verify(old)`-fails assertion proves a real cutover. GitLab CE is too heavy for the VM → MOCK-ONLY. - **cloudflare** (`cloudflare.go`, MOCK-ONLY) — Cloudflare API token *value roll* via `PUT /client/v4/user/tokens/{id}/value` (Bearer). Same token id, fresh value, previous value dead → in-place, `RevokeOld` no-op; keeps id/policies/name stable. Blob `cloudflare:///?id=&token=`. Verify via `/tokens/verify` (`status:active`). - **ghactions** (`ghactions.go`, MOCK-ONLY) — GitHub Actions repo secret. Fetches the repo libsodium public key (`actions/secrets/public-key`), seals a fresh value with `crypto_box_seal` (`golang.org/x/crypto/nacl/box.SealAnonymous`, no new go.mod dep), and `PUT`s it over the existing secret name (overwrite = cutover) → `RevokeOld` no-op. The rotated material is the **secret value** (`val=`), NOT the management `pat=` that authenticates the call. GitHub never returns secret values, so Verify is an existence/authorization check (200 on the secret), not a value match. The emulator owns the box keypair and **decrypts the sealed PUT**, asserting the decrypted value equals the driver's new blob value — proving the encryption is real, not simulated. - **appsecret** (`appsecret.go`, **LIVE-VM**) — LOCAL app signing secret (Django `SECRET_KEY`, Rails `secret_key_base`, JWT/session secrets, …). Generates a fresh 64-hex value and rewrites it **in place across every target file** (atomic temp+rename, mode preserved), keyed on the literal old value so it is file-format-agnostic (.env/yaml/json/toml). In-place ⇒ `RevokeOld` no-op; Verify re-reads every file asserting new-present / old-absent. Blob `appsecret://local/?key=&val=&path=…[&path=…]`. A `minSecretLen` (12) guard refuses to key a replacement on a short/common string (mass-corruption guard), and a zero-replacement result is an error (never claim a rotation that didn't happen). Errors scrub the value via `redactErr`. **appsecret discovery (`env.go`).** An exact-match set of app-signing key names (`SECRET_KEY`, `SECRET_KEY_BASE`, `JWT_SECRET`, `APP_KEY`, …) now tags those `.env` entries `Source="appsecret"` with a self-contained blob carrying the value AND the file path, so the local rotator is driver-ready straight from a scan. Generic API keys (STRIPE_API_KEY) stay plain env secrets. Regression: `scanners_test.go:TestEnvScannerAppSecret`. **appsecret live VM cutover (LIVE-VM).** `lab-provision-appsec.sh` wrote two real config files (`labapp/.env` 0600, `labworker/config.yaml` 0644) sharing one `SECRET_KEY`, seeded the blob into isolated gopass, and ran the shipped binary `incredigo rotate --execute --prefix imported/`. All assertions PASSED: backup gate sealed+verified first; both files rewritten with a fresh 64-hex value; OLD value gone from both; unrelated `PORT=8080` preserved; 0644 mode preserved. The run's table showed `appsecret LIVE-VM` — confirming the new PROOF column renders in the real binary. **Honesty taxonomy (the point of the manifest).** Running in the VM does NOT make a driver LIVE-VM — only the *real target software* validating the cutover does. So this batch is 3× MOCK-ONLY (gitlab/cloudflare/ghactions: no self-host / heavy CE) + 1× LIVE-VM (appsecret: real local files). AWS remains the canonical example: it ran in the VM but only against `moto` (a mock), so it is MOCK-ONLY. Promoting a MOCK-ONLY driver needs only a real service + re-run — zero driver code changes (the drivers contain no test awareness). Full suite after the batch: **82 green**, `-race` clean on rotate/sink/vault. ## Service-API batch 2 — 6 drivers (mongo, npm, gcp, twilio, flyio, k8s), 2026-06-18 Six more rotation drivers, each **real-service code only** (no test awareness; tests inject just an `HTTPClient`/`Bin`), validated against protocol-enforcing emulators: - **mongo** (in-place DB) — `changeUserPassword` over the OLD authenticated connection is itself the cutover; `RevokeOld` is a no-op. NO secret on argv: the URI handed to `mongosh` carries host/db only, both passwords go in on stdin, captured output is scrubbed. Fake `mongosh` stub authenticates against a state-file password → proves old password dies. - **npm** (provider-API create→verify→revoke) — `POST /-/npm/v1/tokens` (Bearer old) → `/-/whoami` (Bearer new) → `DELETE …/tokens/token/`. Blob records the token `key=` for deletion; degrades to guided when absent. - **gcp** (provider-API, service-account KEY) — mints an **RS256 JWT signed with the SA's own private key** (`crypto/rsa`, stdlib, no new dep), exchanges it for an access token, then `CreateServiceAccountKey` → GET serviceAccount (verify) → `DeleteServiceAccountKey`. The emulator **verifies the JWT signature** against the key's public key, so the test proves genuine signing, not a stub. SA JSON round-trips base64-wrapped in the one-line blob. - **twilio** (provider-API) — `POST …/Keys.json` (Basic old) → GET key (Basic new) → `DELETE …/Keys/.json`. Basic-auth username=KeySid, password=secret. - **flyio** (provider-API, GraphQL) — a **managing token** mints/deletes the **deploy token** (the rotated value) via `createLimitedAccessToken` / `deleteLimitedAccessToken`; verify runs the app query as the new deploy token. Same "auth rotates a value" shape as ghactions. - **k8s** (provider-API) — creates a new `kubernetes.io/service-account-token` Secret, waits for the controller to populate `.data.token`, GETs the SA to verify, then DELETEs the old Secret (which invalidates the old token). **All 6 are MOCK-ONLY** — honestly, because they were proven only against emulators. mongo, npm and k8s are LIVE-VM *candidates* (real MongoDB / a throwaway npm registry / a k3s cluster would promote them with zero driver-code change), but per the standing rule a driver is only LIVE-VM after the *real target* validates the cutover, so they stay MOCK-ONLY until those VM POCs run. Each emulator enforces credential validity and the tests assert `Verify(old)` fails after revoke, plus a leak canary (no token/secret/`PRIVATE KEY` in Identity/Source/Location). Full suite after batch 2: **103 green**, `-race` clean on rotate/sink/vault. ## Summary - Harness result: **31/31 PASS**, including the global secret-leak canary (no CANARY token or `PRIVATE KEY` material reached any command output). - One **real defect** found by manual triage of the backup gate, since fixed. ## Defect 1 — backup gate silently under-covered dot-segment entries (FIXED) **Severity:** high — a silent violation of Hard Rule #1 (backup before rotate). **Symptom.** After `migrate`, the store held 10 imported credentials, but the rotate dry-run backup gate sealed/verified only **7**. The 3 missing were the `.env`-derived entries: `imported/env/.env/{api_token,database_url,sendgrid_api_key}`. **Root cause.** `internal/sink/gopass.go:slug()` lower-cased and space-collapsed the credential Identity but **kept leading dots**. An env credential has `Identity = ".env / API_TOKEN"`, so `StorePath` became `imported/env/.env/api_token`. The `.env` path segment is dot-prefixed, and gopass **hides dot-prefixed directories/leaves from `ls --flat`**. `bundle.ListPaths` (and the backup gate that consumes it) enumerate via `ls --flat`, so those 3 entries were invisible — never sealed, never verified. **Why it was silent.** The gate's integrity check compares *sealed == verified* counts. Both were 7, so 7==7 passed. The gate had no notion of *expected* count, so under-coverage produced no error. **Fix.** `slug()` now strips leading dots per `/`-separated path segment, so `.env` → `env` (`imported/env/env/api_token`). Entries are now enumerable and the gate covers all of them. Regression test: `internal/sink/bundle_test.go:TestStorePathNoDotSegment` asserts no produced StorePath contains a dot-prefixed segment. Full suite: 42 tests green, `-race` clean on `sink`/`rotate`. ## Root cause of the *lapse* (why it stayed silent) and prevention The slug bug was the trigger, but it stayed silent because of a deeper flaw in the gate: `rotate.Snapshot` asserts `verified == res.n`, and **both counts descend from the same `gopass ls --flat` enumeration** (`ExportTo` → `ListPaths`, bundle.go). A blind spot in that enumeration is inherited by both sides, so the comparison can never catch it. The check proves *round-trip integrity* (the bundle re-reads to the same count it sealed), not *coverage* (the bundle holds every credential in the store). **A safety gate that compares two values derived from the same source cannot detect an error in that source.** Prevention, layered: 1. **Source fix (done).** `slug()` strips leading dots → incredigo can no longer write any path that `ls --flat` hides. `TestStorePathNoDotSegment` locks it in. This removes the only trigger incredigo controls and is backend-agnostic. 2. **Independent coverage cross-check (done, harness).** `lab-harness.sh` T8b counts the on-disk `*.gpg` files under `imported/` — ground truth that does **not** go through `ls --flat` — and asserts the gate sealed exactly that many. This is where an independent count is cheap and reliable (`GOPASS_HOMEDIR` is known/controlled). It would have failed loudly on the original 7-vs-10. **Production gate — deliberately *not* changed.** Adding the same on-disk `.gpg` walk to `rotate.Snapshot` would couple the safety-critical gate to gopass's fs-backend layout (store-dir resolution varies by gopass version/mount and breaks for non-fs backends). That trades one fragility for another and could make the gate *falsely* abort or pass. The two-layer defense above (write-side invariant + harness cross-check) closes the class without that coupling. If we later want an in-gate coverage assertion, the clean path is a backend-agnostic enumeration primitive in `sink` (not an fs-layout assumption) — a separate design task. ## ICM evaluation (does the methodology help here?) **Recommendation: do not adopt ICM for the incredigo codebase now.** ICM (`icm-scaffold` skill) structures an *LLM-orchestrated* workflow as numbered markdown stages — "folders are the program, one agent reads the right file at the right moment." incredigo is a deterministic **compiled Go tool** with a package architecture, an interface/registry extension model, and a real test suite. ICM's folders-as-program model does not map onto a binary, and the project directive explicitly bans an LLM in the destructive hot path. Adopting ICM here would add scaffolding with no control-flow benefit. The one place it *could* fit later is the **AI-router / browser long-tail** rotation layer (ROTATION.md step 4), which is by design a skill/MCP layer *on top* of the Go core — revisit ICM only when that layer is built.