incredigo/docs/FINDINGS-2026-06-16.md

Findings — controlled-environment test run (2026-06-16)

Source: ~/incredigo-lab-stage/lab-harness.sh (fresh isolated gopass store + fake, canary-tagged inputs for all 8 scanners). FAKE creds only; never touches a real store.

Live rotation POC — first real driver (postgres), 2026-06-16

Result: real end-to-end cutover proven in the incredigo-sbx Multipass VM (real gopasspw/gopass 1.15.16, real PostgreSQL 16, isolated GOPASS_HOMEDIR/GNUPGHOME, FAKE labapp@labdb role).

Flow exercised by the shipped binary via incredigo rotate --execute (gated: INCREDIGO_ALLOW_EXECUTE=1 + --execute):

1. Backup gate sealed + verified 1 entry → ~/.incredigo/rotation-backups/<ts>.age before any change (Hard Rule #1).
2. Rotate ran ALTER USER labapp WITH PASSWORD '<48-hex>' over the OLD connection.
3. Verify(new) → SELECT 1 with the new secret passed.
4. Store the rebuilt DSN back into gopass at the same entry; re-read + Verify(stored) passed.
5. RevokeOld = no-op (in-place ALTER is itself the cutover).

Cutover assertions (all passed):

• OLD password (oldpw835) rejected after rotation — FATAL: password authentication failed.
• NEW 24-byte hex password (read back from gopass) authenticates.
• Audit log (rotate-execute) contains no password substring (old or new).

Spine: internal/rotate/execute.go (Execute), driver internal/rotate/postgres.go (self-registers via init), sourcing cmd/incredigo gopassCredsForExecute (Mode A — reconstructs Source from the gopass path segment), persistence sink.Gopass.Show/InsertAt. Unit proof of the same cutover (no live DB): internal/rotate/postgres_test.go:TestPostgresRotateRealCutover. Full suite: 45 green.

Live rotation POC — gitea PAT driver (provider-API pattern), 2026-06-17

Result: real create-new → verify → revoke-old cutover proven against a real Gitea 1.25.0 (sqlite, http on loopback) inside the incredigo-sbx VM, isolated gopass store, self-owned labadmin account. This is the first provider-API driver and the first whose RevokeOld is a real destructive API call.

Provision: ~/incredigo-lab-stage/lab-provision-gitea.sh (stands up Gitea, mints a seed token via basic auth, seeds the driver-ready blob into gopass). Flow exercised by the shipped binary via incredigo rotate --execute (gated):

1. Backup gate sealed + verified 1 entry before any change (Hard Rule #1).
2. Rotate POST /api/v1/users/labadmin/tokens (basic auth) → new token.
3. Verify(new) GET /api/v1/user with the new token → 200, login=labadmin.
4. Store the rebuilt blob back into gopass; re-read + Verify(stored) passed.
5. RevokeOld DELETE …/tokens/<old-name> — a real token deletion.

Cutover assertions (all passed):

• OLD token → GET /user returns 401 after rotation (revoked).
• NEW token (read back from gopass) → 200, login=labadmin.
• Gitea token list now shows only incredigo-rotated-<ts>; the seed token is gone.
• Audit log (rotate-execute) contains no token substring (old or new); old ≠ new.

Spine: internal/rotate/gitea.go (single-line https://user:token@host/?name=&pw= blob so it round-trips the existing single-line gopass sink; authManage prefers basic auth when a mgmt password is present, else token auth). Discovery for the gitea PAT type: internal/discover/tea.go (Gitea tea CLI config → driver-ready blob, Source gitea). Unit proof of the same enforced cutover (httptest Gitea emulator that 401s a revoked token): internal/rotate/gitea_test.go.

SSH key driver (unit-proven cutover), 2026-06-17

internal/rotate/sshkey.go mints a new ed25519 keypair entirely in memory (golang.org/x/crypto/ssh; private key never touches disk — Hard Rule #3), installs only the new PUBLIC key into authorized_keys, verifies by SSH-dialing the authority, and RevokeOld removes the old public key (atomic rewrite). Proven by an in-process SSH server test: old key rejected after revoke, new key works, no PRIVATE KEY material in authorized_keys (internal/rotate/sshkey_test.go). Discovery broadened (internal/discover/ssh.go): content-based key detection (not just id_*) + ~/.ssh/config IdentityFile parsing. Live VM POC not yet run.

Full suite after all three drivers: 50 green, -race clean on rotate/sink/vault/discover.

Live rotation POC — AWS IAM access key driver (cloud-key pattern), 2026-06-17

Result: real create-new → verify → revoke-old cutover proven against a self-hosted moto 5.2.2 mock AWS (moto_server on http://127.0.0.1:5000) inside the incredigo-sbx VM, isolated gopass store, self-created IAM user labapp. This is the cleanest provider-API driver: the OLD credential self-identifies — an access key's AccessKeyId is non-secret and rides in the secret blob, so RevokeOld deletes exactly the right key (no name-capture dance Gitea needed).

Implementation note: the driver speaks the AWS Query protocol (form-POST Action=…&Version=…) signed with hand-rolled SigV4 (crypto/hmac) — zero new go.mod deps, keeping the static binary lean (the AWS SDK v2 would have added ~50 indirect deps). Spine: internal/rotate/aws.go. Secret blob is the single line aws://<AccessKeyId>:<SecretAccessKey>@aws/?region=<r>[&endpoint=<url>] so it round-trips the single-line gopass sink; endpoint= overrides the API base for moto/LocalStack, absent → real iam/sts.amazonaws.com. Discovery broadened: internal/discover/aws.go now emits that driver-ready blob (was the bare secret) and resolves region from env / ~/.aws/config.

Flow exercised by the shipped binary via incredigo rotate --execute (gated):

1. Backup gate sealed + verified 1 entry before any change (Hard Rule #1).
2. Rotate IAM CreateAccessKey (no UserName → moto resolves the calling identity from the signing key) → new key pair.
3. Verify(new) STS GetCallerIdentity with the new key → ARN returned.
4. Store the rebuilt blob into gopass; re-read + Verify(stored) passed.
5. RevokeOld IAM DeleteAccessKey targeting the OLD AccessKeyId — a real delete.

Cutover assertions (all passed, lab-verify-aws.sh):

• After rotate, IAM ListAccessKeys(labapp) shows only the new key; the old AccessKeyId is gone at the provider. old ≠ new.
• New key (read back from gopass) authenticates; audit log (rotate-execute) carries no AccessKeyId and no SecretAccessKey.

moto enforcement caveat (investigated, important). moto's STS GetCallerIdentity does not validate access-key existence — a deleted key still returns a (generic user/moto) ARN rather than a 403. So against moto, the driver's STS-based Verify cannot itself prove "old key dead"; the live cutover proof is therefore IAM ListAccessKeys (ground truth that the delete took effect), asserted in lab-verify-aws.sh. The enforced 403-on-revoked-key path (what real AWS does) is proven instead by the unit test: internal/rotate/aws_test.go's awsEmu reads the AccessKeyId from the SigV4 Credential= header and rejects any key not currently present, so TestAWSRotateRealCutover asserts the old key returns 403 after revoke, the new key still authenticates, and old ≠ new. The driver's Verify (plain "ARN non-empty") is correct for real AWS, which does 403 an invalid key; no moto-specific leniency was baked into the driver.

Provision: ~/incredigo-lab-stage/lab-provision-aws.sh (starts moto, creates labapp + seed key via boto3, seeds the driver-ready blob into isolated gopass); verify: ~/incredigo-lab-stage/lab-verify-aws.sh. Unit proofs: TestAWSRotateRealCutover (enforced cutover), TestAWSSecretRoundTrip (blob carries /+= secrets intact), TestSigV4HeadersStable (Authorization shape).

Full suite after four drivers: 53 green, -race clean on rotate/sink/vault/discover.

VPN / router / DB-clone batch — 5 drivers, 3 live cutovers, 2026-06-18

Approved expansion (VPN + router secondary persona, plus Tier-1 DB clones). Five new one-file drivers, each with a table-driven unit test (cutover proof + leak-check). Three were additionally proven live in the incredigo-sbx VM against real services; one is unit-proven with a live QEMU POC deferred; one is unit-only by design.

Driver	Pattern	Live POC	Proof
mysql	in-place DB password	PASS (real MariaDB)	lab-provision-dbclones.sh
redis	in-place CONFIG SET requirepass	PASS (real redis-server)	lab-provision-dbclones.sh
wireguard	local keypair regen + peer propagate	PASS (real wg)	lab-provision-wg.sh
openwrt	in-place SSH passwd	deferred (QEMU on 1-CPU VM)	unit: in-process SSH server
mullvad	provider-API device key	n/a (needs paid acct)	unit: httptest TLS emulator

MySQL + Redis live cutover (lab-provision-dbclones.sh, all assertions PASS). Real mariadb-server + redis-server, isolated gopass, fake labapp@labdb and a fake redis requirepass. Backup gate sealed+verified first; rotate --execute flipped both; the OLD password is rejected and the NEW one (read back via gopass show -o) authenticates for each engine.

Driver gap found & fixed — MariaDB unprivileged self-rotation. MariaDB rejects ALTER USER CURRENT_USER() IDENTIFIED BY … for a non-admin (ERROR 1227 — it demands the global CREATE USER privilege even for one's own account), which is exactly the target persona's situation (a vibe coder's DB user is scoped to one schema with GRANT ALL ON app.*). The MySQL-only USER() spelling also fails on MariaDB (ERROR 1064). Fix (internal/rotate/mysql.go): try a fallback chain — ALTER USER CURRENT_USER() (MySQL self-service) → SET PASSWORD = PASSWORD('…') (MariaDB / MySQL 5.7) → SET PASSWORD = '…' (MySQL 8 / MariaDB ≥10.4); first to succeed wins. After the fix the live POC passed on MariaDB.

WireGuard live cutover (lab-provision-wg.sh, all assertions PASS). Real wireguard-tools: wg0 = ours, wg1 = the peer that trusts us. Seeds wg0's old private key into isolated gopass, rotate --execute mints a new clamped curve25519 keypair, applies the new private key to wg0 (via stdin → wg set wg0 private-key /dev/stdin, never argv), and re-points the peer to the new public key (carrying over allowed-ips). Assertions: private key changed; peer trusts NEW pubkey; peer dropped OLD pubkey; and the headline compatibility proof — wg0's live interface public key equals the public key our Go code derived, i.e. our in-process curve25519 derivation matches real wg pubkey.

POC scripting gotcha (Mode A prefix). Mode A reconstructs the driver Source from the first path segment after --prefix (imported/<source>/<slug>). The WG POC first ran with --prefix imported/wireguard/, which made Source="wg0" (no driver Detected it → ROTATED 0). Fix: point --prefix at the imported/ root so imported/wireguard/wg0 resolves Source="wireguard". This is a harness fix, not a driver bug, but worth recording: narrowing the prefix below the source dir breaks source recovery.

OpenWrt (unit-proven, live deferred). internal/rotate/openwrt.go dials the router with the OLD password and runs BusyBox passwd, feeding the new password twice on the SSH-encrypted stdin (never argv); setting the password IS the cutover (RevokeOld no-op). Proven by an in-process SSH server (openwrt_test.go) with a mutable current password: old password stops authenticating after rotate, new works, no secret in argv or error strings. A live OpenWrt-in-QEMU POC is the heaviest (full-system emulation on the 1-CPU sandbox VM) and is deferred.

Mullvad (unit-only by design). internal/rotate/mullvad.go is the provider-API device-key pattern: fetch an access token from the account number, POST /accounts/v1/devices {pubkey} to register a freshly-derived WireGuard key, verify the device lists that exact pubkey, then DELETE the old device. Proven against an httptest TLS emulator (mullvad_test.go) that inspects every request body for the private key (never sent) and 404s a deleted device. No live POC — that needs a real paid Mullvad account; the API contract is faithfully emulated instead.

Discovery broadened (Hard-rule standing directive — dynamic discovery per type). The env scanner (internal/discover/env.go) now tags DB connection URLs by scheme: postgres/postgresql → postgres, mysql/mariadb → mysql, redis/rediss → redis (requires a user:password authority), so DATABASE_URL/REDIS_URL/MYSQL_URL in a .env route straight to the matching in-place driver, kept regardless of the secret-name heuristic. Regression: scanners_test.go:TestEnvScanner.

Full suite after the batch: 68 green, -race clean on rotate/sink/vault/discover.

SaaS-token / app-signing batch — 4 drivers + proof manifest, 2026-06-18

Four more drivers, each one-file + table-driven test (cutover proof + leak-check). User direction this batch: keep real-rotation code unmistakably distinct from mock code, and record the real-vs-mock distinction as DATA, not behaviour. Implemented as a single Go table (internal/rotate/proofs.go) + a human mirror (docs/ROTATION-PROOFS.md), surfaced as a PROOF column in incredigo rotate so a mock-only driver can never be mistaken for a live one.

• gitlab (gitlab.go, MOCK-ONLY) — GitLab PAT via POST /api/v4/personal_access_tokens/self/rotate (PRIVATE-TOKEN header). GitLab mints the new token and revokes the caller's old one in the same call → in-place, RevokeOld no-op. Blob gitlab://<host>/?tok=<token>. The httptest emulator enforces the revoke, so the unit test's Verify(old)-fails assertion proves a real cutover. GitLab CE is too heavy for the VM → MOCK-ONLY.
• cloudflare (cloudflare.go, MOCK-ONLY) — Cloudflare API token value roll via PUT /client/v4/user/tokens/{id}/value (Bearer). Same token id, fresh value, previous value dead → in-place, RevokeOld no-op; keeps id/policies/name stable. Blob cloudflare://<host>/?id=<id>&token=<val>. Verify via /tokens/verify (status:active).
• ghactions (ghactions.go, MOCK-ONLY) — GitHub Actions repo secret. Fetches the repo libsodium public key (actions/secrets/public-key), seals a fresh value with crypto_box_seal (golang.org/x/crypto/nacl/box.SealAnonymous, no new go.mod dep), and PUTs it over the existing secret name (overwrite = cutover) → RevokeOld no-op. The rotated material is the secret value (val=), NOT the management pat= that authenticates the call. GitHub never returns secret values, so Verify is an existence/authorization check (200 on the secret), not a value match. The emulator owns the box keypair and decrypts the sealed PUT, asserting the decrypted value equals the driver's new blob value — proving the encryption is real, not simulated.
• appsecret (appsecret.go, LIVE-VM) — LOCAL app signing secret (Django SECRET_KEY, Rails secret_key_base, JWT/session secrets, …). Generates a fresh 64-hex value and rewrites it in place across every target file (atomic temp+rename, mode preserved), keyed on the literal old value so it is file-format-agnostic (.env/yaml/json/toml). In-place ⇒ RevokeOld no-op; Verify re-reads every file asserting new-present / old-absent. Blob appsecret://local/?key=&val=&path=…[&path=…]. A minSecretLen (12) guard refuses to key a replacement on a short/common string (mass-corruption guard), and a zero-replacement result is an error (never claim a rotation that didn't happen). Errors scrub the value via redactErr.

appsecret discovery (env.go). An exact-match set of app-signing key names (SECRET_KEY, SECRET_KEY_BASE, JWT_SECRET, APP_KEY, …) now tags those .env entries Source="appsecret" with a self-contained blob carrying the value AND the file path, so the local rotator is driver-ready straight from a scan. Generic API keys (STRIPE_API_KEY) stay plain env secrets. Regression: scanners_test.go:TestEnvScannerAppSecret.

appsecret live VM cutover (LIVE-VM). lab-provision-appsec.sh wrote two real config files (labapp/.env 0600, labworker/config.yaml 0644) sharing one SECRET_KEY, seeded the blob into isolated gopass, and ran the shipped binary incredigo rotate --execute --prefix imported/. All assertions PASSED: backup gate sealed+verified first; both files rewritten with a fresh 64-hex value; OLD value gone from both; unrelated PORT=8080 preserved; 0644 mode preserved. The run's table showed appsecret LIVE-VM — confirming the new PROOF column renders in the real binary.

Honesty taxonomy (the point of the manifest). Running in the VM does NOT make a driver LIVE-VM — only the real target software validating the cutover does. So this batch is 3× MOCK-ONLY (gitlab/cloudflare/ghactions: no self-host / heavy CE) + 1× LIVE-VM (appsecret: real local files). AWS remains the canonical example: it ran in the VM but only against moto (a mock), so it is MOCK-ONLY. Promoting a MOCK-ONLY driver needs only a real service + re-run — zero driver code changes (the drivers contain no test awareness).

Full suite after the batch: 82 green, -race clean on rotate/sink/vault.

Service-API batch 2 — 6 drivers (mongo, npm, gcp, twilio, flyio, k8s), 2026-06-18

Six more rotation drivers, each real-service code only (no test awareness; tests inject just an HTTPClient/Bin), validated against protocol-enforcing emulators:

• mongo (in-place DB) — changeUserPassword over the OLD authenticated connection is itself the cutover; RevokeOld is a no-op. NO secret on argv: the URI handed to mongosh carries host/db only, both passwords go in on stdin, captured output is scrubbed. Fake mongosh stub authenticates against a state-file password → proves old password dies.
• npm (provider-API create→verify→revoke) — POST /-/npm/v1/tokens (Bearer old) → /-/whoami (Bearer new) → DELETE …/tokens/token/<key>. Blob records the token key= for deletion; degrades to guided when absent.
• gcp (provider-API, service-account KEY) — mints an RS256 JWT signed with the SA's own private key (crypto/rsa, stdlib, no new dep), exchanges it for an access token, then CreateServiceAccountKey → GET serviceAccount (verify) → DeleteServiceAccountKey. The emulator verifies the JWT signature against the key's public key, so the test proves genuine signing, not a stub. SA JSON round-trips base64-wrapped in the one-line blob.
• twilio (provider-API) — POST …/Keys.json (Basic old) → GET key (Basic new) → DELETE …/Keys/<sid>.json. Basic-auth username=KeySid, password=secret.
• flyio (provider-API, GraphQL) — a managing token mints/deletes the deploy token (the rotated value) via createLimitedAccessToken / deleteLimitedAccessToken; verify runs the app query as the new deploy token. Same "auth rotates a value" shape as ghactions.
• k8s (provider-API) — creates a new kubernetes.io/service-account-token Secret, waits for the controller to populate .data.token, GETs the SA to verify, then DELETEs the old Secret (which invalidates the old token).

Each emulator enforces credential validity and the tests assert Verify(old) fails after revoke, plus a leak canary (no token/secret/PRIVATE KEY in Identity/Source/Location).

Live-VM POCs run (2026-06-18/19) → mongo + k8s promoted to LIVE-VM

• mongo → LIVE-VM (lab-provision-mongo.sh): real mongod 8.0 from the official apt repo; labapp (admin, userAdminAnyDatabase) rotates its own password; old password rejected, new works. Surfaced by the real target: real mongosh echoes a labdb> prompt to stdout on a pipe and db.auth returns an object not a bool in 2.8 — the fake-mongosh stub did neither, so the assertion had to filter output to the AUTH-OK/AUTH-FAIL token.
• k8s → LIVE-VM (lab-provision-k8s.sh): real k3s v1.35 kube-apiserver + token controller; labsa (RBAC: create/delete/get Secrets + get its SA) rotates its legacy token Secret; old token → 401, new → 200. Surfaced by the real target: (a) the apiserver serves a self-signed cert, so the driver needed CA-pinned TLS — added ca= (base64 PEM) + an insecure=1 lab-only escape hatch to the blob; (b) the apiserver caches successful auths ~10s (cachedTokenAuthenticator), so the deleted token kept working until the cache expired (~7s) — the POC polls past it. Both are real-deployment config, not test-awareness.
• npm stays MOCK-ONLY (decided, not pending): registry.npmjs.org isn't self-hostable and real POST /-/npm/v1/tokens requires the account password in the body (the driver sends only Bearer old-token). Documented as a contract gap in docs/ROTATION-PROOFS.md rather than faking a LIVE-VM. gcp/twilio/flyio remain MOCK-ONLY (no self-host).

Full suite after batch 2 + POCs: 104 green, -race clean on rotate/sink/vault.

Summary

• Harness result: 31/31 PASS, including the global secret-leak canary (no CANARY token or PRIVATE KEY material reached any command output).
• One real defect found by manual triage of the backup gate, since fixed.

Defect 1 — backup gate silently under-covered dot-segment entries (FIXED)

Severity: high — a silent violation of Hard Rule #1 (backup before rotate).

Symptom. After migrate, the store held 10 imported credentials, but the rotate dry-run backup gate sealed/verified only 7. The 3 missing were the .env-derived entries: imported/env/.env/{api_token,database_url,sendgrid_api_key}.

Root cause. internal/sink/gopass.go:slug() lower-cased and space-collapsed the credential Identity but kept leading dots. An env credential has Identity = ".env / API_TOKEN", so StorePath became imported/env/.env/api_token. The .env path segment is dot-prefixed, and gopass hides dot-prefixed directories/leaves from ls --flat. bundle.ListPaths (and the backup gate that consumes it) enumerate via ls --flat, so those 3 entries were invisible — never sealed, never verified.

Why it was silent. The gate's integrity check compares sealed == verified counts. Both were 7, so 7==7 passed. The gate had no notion of expected count, so under-coverage produced no error.

Fix. slug() now strips leading dots per /-separated path segment, so .env → env (imported/env/env/api_token). Entries are now enumerable and the gate covers all of them. Regression test: internal/sink/bundle_test.go:TestStorePathNoDotSegment asserts no produced StorePath contains a dot-prefixed segment. Full suite: 42 tests green, -race clean on sink/rotate.

Root cause of the lapse (why it stayed silent) and prevention

The slug bug was the trigger, but it stayed silent because of a deeper flaw in the gate: rotate.Snapshot asserts verified == res.n, and both counts descend from the same gopass ls --flat enumeration (ExportTo → ListPaths, bundle.go). A blind spot in that enumeration is inherited by both sides, so the comparison can never catch it. The check proves round-trip integrity (the bundle re-reads to the same count it sealed), not coverage (the bundle holds every credential in the store). A safety gate that compares two values derived from the same source cannot detect an error in that source.

Prevention, layered:

1. Source fix (done). slug() strips leading dots → incredigo can no longer write any path that ls --flat hides. TestStorePathNoDotSegment locks it in. This removes the only trigger incredigo controls and is backend-agnostic.
2. Independent coverage cross-check (done, harness). lab-harness.sh T8b counts the on-disk *.gpg files under imported/ — ground truth that does not go through ls --flat — and asserts the gate sealed exactly that many. This is where an independent count is cheap and reliable (GOPASS_HOMEDIR is known/controlled). It would have failed loudly on the original 7-vs-10.

Production gate — deliberately not changed. Adding the same on-disk .gpg walk to rotate.Snapshot would couple the safety-critical gate to gopass's fs-backend layout (store-dir resolution varies by gopass version/mount and breaks for non-fs backends). That trades one fragility for another and could make the gate falsely abort or pass. The two-layer defense above (write-side invariant + harness cross-check) closes the class without that coupling. If we later want an in-gate coverage assertion, the clean path is a backend-agnostic enumeration primitive in sink (not an fs-layout assumption) — a separate design task.

ICM evaluation (does the methodology help here?)

Recommendation: do not adopt ICM for the incredigo codebase now. ICM (icm-scaffold skill) structures an LLM-orchestrated workflow as numbered markdown stages — "folders are the program, one agent reads the right file at the right moment." incredigo is a deterministic compiled Go tool with a package architecture, an interface/registry extension model, and a real test suite. ICM's folders-as-program model does not map onto a binary, and the project directive explicitly bans an LLM in the destructive hot path. Adopting ICM here would add scaffolding with no control-flow benefit. The one place it could fit later is the AI-router / browser long-tail rotation layer (ROTATION.md step 4), which is by design a skill/MCP layer on top of the Go core — revisit ICM only when that layer is built.