rotate: promote k8s + mongo to LIVE-VM via real-target POCs

Ran live cutover POCs in the sandbox VM against the real target software and
promoted both drivers from MOCK-ONLY to LIVE-VM in the proof table (data, not
driver behaviour):

- k8s: proven against real k3s v1.35 kube-apiserver + token controller
  (lab-provision-k8s.sh). The real apiserver serves a self-signed cert, so the
  driver gained CA-pinned TLS — blob params ca= (base64 PEM, pinned as the only
  trusted root) and insecure=1 (lab-only escape hatch), routed through a new
  clientFor(cr). Neither is test-awareness; a real deployment configures the same
  CA. Added TestK8sTLSTrust covering CA-pinned / insecure / untrusted-rejection
  and the blob round-trip.
- mongo: proven against real mongod 8.0 (lab-provision-mongo.sh).

npm stays MOCK-ONLY by decision (registry.npmjs.org not self-hostable; real
create-token requires the account password in the body) — documented as a
contract gap rather than faking a LIVE-VM. gcp/twilio/flyio remain MOCK-ONLY.

Live POCs surfaced real-target-only behaviour now recorded in the docs: the
apiserver's ~10s successful-auth cache (old token kept working ~7s after Secret
deletion) and real mongosh's stdout prompt. Full suite 104 green, -race clean.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

docs/FINDINGS-2026-06-16.md

@@ -290,14 +290,30 @@ just an `HTTPClient`/`Bin`), validated against protocol-enforcing emulators:
   for the controller to populate `.data.token`, GETs the SA to verify, then DELETEs the old
   Secret (which invalidates the old token).
 
-**All 6 are MOCK-ONLY** — honestly, because they were proven only against emulators. mongo,
-npm and k8s are LIVE-VM *candidates* (real MongoDB / a throwaway npm registry / a k3s cluster
-would promote them with zero driver-code change), but per the standing rule a driver is only
-LIVE-VM after the *real target* validates the cutover, so they stay MOCK-ONLY until those VM
-POCs run. Each emulator enforces credential validity and the tests assert `Verify(old)` fails
+Each emulator enforces credential validity and the tests assert `Verify(old)` fails
 after revoke, plus a leak canary (no token/secret/`PRIVATE KEY` in Identity/Source/Location).
 
-Full suite after batch 2: **103 green**, `-race` clean on rotate/sink/vault.
+### Live-VM POCs run (2026-06-18/19) → mongo + k8s promoted to LIVE-VM
+
+- **mongo → LIVE-VM** (`lab-provision-mongo.sh`): real `mongod` 8.0 from the official apt
+  repo; `labapp` (admin, `userAdminAnyDatabase`) rotates its own password; old password
+  rejected, new works. *Surfaced by the real target:* real `mongosh` echoes a `labdb> ` prompt
+  to stdout on a pipe and `db.auth` returns an object not a bool in 2.8 — the fake-mongosh stub
+  did neither, so the assertion had to filter output to the `AUTH-OK`/`AUTH-FAIL` token.
+- **k8s → LIVE-VM** (`lab-provision-k8s.sh`): real **k3s v1.35** kube-apiserver + token
+  controller; `labsa` (RBAC: create/delete/get Secrets + get its SA) rotates its legacy
+  token Secret; old token → `401`, new → `200`. *Surfaced by the real target:* (a) the
+  apiserver serves a self-signed cert, so the driver needed CA-pinned TLS — added `ca=`
+  (base64 PEM) + an `insecure=1` lab-only escape hatch to the blob; (b) the apiserver caches
+  *successful* auths ~10s (`cachedTokenAuthenticator`), so the deleted token kept working until
+  the cache expired (~7s) — the POC polls past it. Both are real-deployment config, not
+  test-awareness.
+- **npm stays MOCK-ONLY** (decided, not pending): `registry.npmjs.org` isn't self-hostable and
+  real `POST /-/npm/v1/tokens` requires the **account password in the body** (the driver sends
+  only Bearer old-token). Documented as a contract gap in `docs/ROTATION-PROOFS.md` rather than
+  faking a LIVE-VM. gcp/twilio/flyio remain MOCK-ONLY (no self-host).
+
+Full suite after batch 2 + POCs: **104 green**, `-race` clean on rotate/sink/vault.
 
 ## Summary
 

docs/ROTATION-PROOFS.md

@@ -41,6 +41,17 @@ stops a mock-only driver from ever being mistaken for a live one.
 | `wireguard` | real `wg` | `lab-provision-wg.sh` |
 | `gitea` | real Gitea | `lab-provision-gitea.sh` |
 | `appsecret` | real local files | `lab-provision-appsec.sh` |
+| `mongo` | real `mongod` 8.0 | `lab-provision-mongo.sh` |
+| `k8s` | real k3s `v1.35` kube-apiserver | `lab-provision-k8s.sh` |
+
+> **k8s live-POC note (auth-cache timing):** the real kube-apiserver caches *successful*
+> token authentications for ~10s (`cachedTokenAuthenticator`). After `RevokeOld` deletes the
+> legacy token Secret, the old token keeps authenticating until that cache entry expires — so
+> the POC polls the old token for up to 30s and asserts it flips to `401` (it does, ~7s). The
+> `httptest` emulator invalidates instantly, so this delay only surfaced against real k3s —
+> exactly what a live POC is for. The driver needed real-CA TLS to talk to the apiserver, so
+> the blob gained `ca=` (base64 PEM, pinned as the only trusted root) and an `insecure=1`
+> lab-only escape hatch; neither is test-awareness — a real deployment configures the same CA.
 
 ### MOCK-ONLY — proven only against an emulator / mock
 
@@ -53,12 +64,10 @@ stops a mock-only driver from ever being mistaken for a live one.
 | `cloudflare` | `httptest` emulator | no self-host |
 | `ghactions` | `httptest` emulator (holds the libsodium box keypair, decrypts the sealed PUT) | no self-host |
 | `gitlab` | `httptest` emulator | GitLab CE too heavy for the VM |
-| `mongo` | fake-`mongosh` shell stub (auths against a state file) | real-MongoDB VM POC not yet run |
-| `npm` | `httptest` registry emulator | real registry never hit |
+| `npm` | `httptest` registry emulator | real registry never hit (see caveat below) |
 | `gcp` | `httptest` IAM emulator (verifies the driver's RS256 JWT against the key's public key) | no self-host |
 | `twilio` | `httptest` emulator (Basic-auth Keys resource) | no self-host |
 | `flyio` | `httptest` GraphQL emulator | no self-host |
-| `k8s` | `httptest` apiserver emulator (token-Secret create/delete) | real k3s POC not yet run |
 
 ## What the MOCK-ONLY emulators actually prove
 
@@ -73,7 +82,7 @@ protocol so the test exercises the driver's true cutover logic:
   on `PUT` **decrypts the sealed value** and stores the plaintext; the test asserts the
   decrypted value equals the new value the driver put in the rebuilt blob — proving the
   sealed-box encryption is correct, not simulated.
-- **npm / twilio / k8s** — each emulator enforces credential validity (a token/key
+- **npm / twilio** — each emulator enforces credential validity (a token/key
   authenticates only while it exists) and the test asserts `Verify(old)` fails after
   `RevokeOld` — proving the create→verify→revoke cutover really happened, old credential
   dead, new one alive.
@@ -85,10 +94,17 @@ protocol so the test exercises the driver's true cutover logic:
 - **flyio** — a managing token mints/deletes the deploy token over GraphQL; the deploy token
   authenticates the app query only while it exists, so the test proves the managing-token →
   deploy-token rotation and that the revoked deploy token stops working.
-- **mongo** — the fake `mongosh` authenticates `db.auth` against a password held in a state
-  file and only `changeUserPassword` updates it; the test proves the in-place cutover (old
-  password rejected after rotation). No password ever reaches `mongosh`'s argv.
 
-Promoting any of these to LIVE-VM only requires standing up the real service (a self-hosted
-GitLab CE / a real Cloudflare token / a throwaway GitHub repo / a real MongoDB / a real npm
-registry / a k3s cluster) and re-running the same driver against it — no driver code changes.
+> **npm caveat (why it is not just "pending a VM run"):** the public `registry.npmjs.org`
+> is not self-hostable, and the classic `POST /-/npm/v1/tokens` create-token call on real
+> npm **requires the account password in the request body** (it is the non-interactive form
+> of `npm token create`, which prompts for your password). The driver authenticates the
+> create with the *old token* as Bearer only, which the emulator accepts but real npmjs does
+> not. So `npm` stays MOCK-ONLY pending either (a) a self-hosted registry that implements the
+> token API faithfully, or (b) adding password-in-body support to the create call. This is
+> exactly the kind of contract gap a live-POC attempt is meant to surface.
+
+Promoting the rest to LIVE-VM only requires standing up the real service (a self-hosted
+GitLab CE / a real Cloudflare token / a throwaway GitHub repo) and re-running the same
+driver against it — no driver code changes. (`k8s` and `mongo` have already made this jump;
+see the LIVE-VM table above.)

internal/rotate/k8s.go

@@ -3,6 +3,8 @@ package rotate
 import (
 	"bytes"
 	"context"
+	"crypto/tls"
+	"crypto/x509"
 	"encoding/base64"
 	"encoding/json"
 	"fmt"
@@ -32,6 +34,10 @@ import (
 //   - sa=      the ServiceAccount name (the new Secret is annotated to it).
 //   - secret=  the token Secret's name — required to DELETE the old token.
 //   - token=   the bearer token value (rotated).
+//   - ca=      OPTIONAL base64(std) of the cluster CA PEM bundle. A real apiserver
+//              serves a self-signed cert, so the driver pins this CA as the only
+//              trusted root for the TLS handshake. Omit for a loopback emulator.
+//   - insecure=1  OPTIONAL escape hatch: skip TLS verification entirely (lab only).
 //
 // AUTH: every call authenticates with a token Bearer header against the apiserver. The
 // rotating SA needs RBAC to create/delete Secrets in its namespace. The new token is
@@ -53,20 +59,44 @@ func (k *K8s) Name() string { return "k8s" }
 // Detect claims credentials tagged Source == "k8s".
 func (k *K8s) Detect(c discover.Credential) bool { return c.Source == "k8s" }
 
-func (k *K8s) client() *http.Client {
+// clientFor returns the HTTP client for a given credential. An injected client
+// (tests / explicit CA config) always wins. Otherwise it builds a 15s client whose
+// TLS trust is pinned to the credential's ca= bundle (a real apiserver's self-signed
+// cert), or with verification disabled when insecure=1 is set (lab only).
+func (k *K8s) clientFor(cr k8sCred) (*http.Client, error) {
 	if k.HTTPClient != nil {
-		return k.HTTPClient
+		return k.HTTPClient, nil
 	}
-	return &http.Client{Timeout: 15 * time.Second}
+	tlsCfg := &tls.Config{MinVersion: tls.VersionTLS12}
+	switch {
+	case cr.insecure:
+		tlsCfg.InsecureSkipVerify = true
+	case cr.ca != "":
+		pem, err := base64.StdEncoding.DecodeString(cr.ca)
+		if err != nil {
+			return nil, fmt.Errorf("k8s: decode ca bundle: %w", err)
+		}
+		pool := x509.NewCertPool()
+		if !pool.AppendCertsFromPEM(pem) {
+			return nil, fmt.Errorf("k8s: ca bundle contained no usable certificates")
+		}
+		tlsCfg.RootCAs = pool
+	}
+	return &http.Client{
+		Timeout:   15 * time.Second,
+		Transport: &http.Transport{TLSClientConfig: tlsCfg},
+	}, nil
 }
 
 // k8sCred is the parsed credential blob. None of its fields are ever logged.
 type k8sCred struct {
-	base   string // scheme://host:port
-	ns     string
-	sa     string
-	secret string // token Secret name
-	token  string
+	base     string // scheme://host:port
+	ns       string
+	sa       string
+	secret   string // token Secret name
+	token    string
+	ca       string // base64(std) cluster CA PEM bundle (optional)
+	insecure bool   // skip TLS verification (lab only)
 }
 
 func parseK8sCred(v *vault.Vault, h *vault.Handle) (k8sCred, error) {
@@ -89,7 +119,7 @@ func parseK8sCred(v *vault.Vault, h *vault.Handle) (k8sCred, error) {
 		return k8sCred{}, fmt.Errorf("k8s: secret has no apiserver host")
 	}
 	q := u.Query()
-	c := k8sCred{base: scheme + "://" + u.Host, ns: q.Get("ns"), sa: q.Get("sa"), secret: q.Get("secret"), token: q.Get("token")}
+	c := k8sCred{base: scheme + "://" + u.Host, ns: q.Get("ns"), sa: q.Get("sa"), secret: q.Get("secret"), token: q.Get("token"), ca: q.Get("ca"), insecure: q.Get("insecure") == "1"}
 	if c.ns == "" || c.sa == "" {
 		return k8sCred{}, fmt.Errorf("k8s: secret missing namespace or serviceaccount")
 	}
@@ -110,6 +140,12 @@ func (c k8sCred) build() string {
 		q.Set("secret", c.secret)
 	}
 	q.Set("token", c.token)
+	if c.ca != "" {
+		q.Set("ca", c.ca)
+	}
+	if c.insecure {
+		q.Set("insecure", "1")
+	}
 	u.RawQuery = q.Encode()
 	return u.String()
 }
@@ -132,8 +168,9 @@ type k8sSecretObj struct {
 	Data map[string]string `json:"data,omitempty"`
 }
 
-// do issues an authenticated apiserver request and returns the body for 2xx responses.
-func (k *K8s) do(ctx context.Context, method, urlStr, token string, body []byte) ([]byte, int, error) {
+// do issues an authenticated apiserver request and returns the body. The credential
+// carries the TLS trust config (ca=/insecure=) used to build the client.
+func (k *K8s) do(ctx context.Context, cr k8sCred, method, urlStr string, body []byte) ([]byte, int, error) {
 	var r io.Reader
 	if body != nil {
 		r = bytes.NewReader(body)
@@ -142,11 +179,15 @@ func (k *K8s) do(ctx context.Context, method, urlStr, token string, body []byte)
 	if err != nil {
 		return nil, 0, err
 	}
-	req.Header.Set("Authorization", "Bearer "+token)
+	req.Header.Set("Authorization", "Bearer "+cr.token)
 	if body != nil {
 		req.Header.Set("Content-Type", "application/json")
 	}
-	resp, err := k.client().Do(req)
+	cl, err := k.clientFor(cr)
+	if err != nil {
+		return nil, 0, err
+	}
+	resp, err := cl.Do(req)
 	if err != nil {
 		return nil, 0, err
 	}
@@ -181,7 +222,7 @@ func (k *K8s) Rotate(ctx context.Context, c discover.Credential, v *vault.Vault)
 	obj.Metadata.Annotations = map[string]string{"kubernetes.io/service-account.name": cr.sa}
 	body, _ := json.Marshal(obj)
 
-	_, code, err := k.do(ctx, http.MethodPost, cr.secretsURL(""), cr.token, body)
+	_, code, err := k.do(ctx, cr, http.MethodPost, cr.secretsURL(""), body)
 	if err != nil {
 		return nil, fmt.Errorf("k8s: create token secret: %w", err)
 	}
@@ -203,7 +244,7 @@ func (k *K8s) Rotate(ctx context.Context, c discover.Credential, v *vault.Vault)
 // awaitToken polls the new Secret until its .data.token is present, decoding it.
 func (k *K8s) awaitToken(ctx context.Context, cr k8sCred, name string) (string, error) {
 	for attempt := 0; attempt < 10; attempt++ {
-		rb, code, err := k.do(ctx, http.MethodGet, cr.secretsURL(name), cr.token, nil)
+		rb, code, err := k.do(ctx, cr, http.MethodGet, cr.secretsURL(name), nil)
 		if err != nil {
 			return "", fmt.Errorf("k8s: read token secret: %w", err)
 		}
@@ -236,7 +277,7 @@ func (k *K8s) Verify(ctx context.Context, newSecret *vault.Handle, v *vault.Vaul
 		return err
 	}
 	saURL := cr.base + "/api/v1/namespaces/" + url.PathEscape(cr.ns) + "/serviceaccounts/" + url.PathEscape(cr.sa)
-	_, code, err := k.do(ctx, http.MethodGet, saURL, cr.token, nil)
+	_, code, err := k.do(ctx, cr, http.MethodGet, saURL, nil)
 	if err != nil {
 		return fmt.Errorf("k8s verify: %w", err)
 	}
@@ -257,7 +298,7 @@ func (k *K8s) RevokeOld(ctx context.Context, c discover.Credential, v *vault.Vau
 	if cr.secret == "" {
 		return fmt.Errorf("k8s revoke: old token has no recorded secret name — delete it manually (guided)")
 	}
-	_, code, err := k.do(ctx, http.MethodDelete, cr.secretsURL(cr.secret), cr.token, nil)
+	_, code, err := k.do(ctx, cr, http.MethodDelete, cr.secretsURL(cr.secret), nil)
 	if err != nil {
 		return fmt.Errorf("k8s revoke: %w", err)
 	}

internal/rotate/k8s_test.go

@@ -6,8 +6,10 @@ import (
 	"encoding/base64"
 	"encoding/hex"
 	"encoding/json"
+	"encoding/pem"
 	"net/http"
 	"net/http/httptest"
+	"net/url"
 	"strings"
 	"sync"
 	"testing"
@@ -218,6 +220,56 @@ func TestK8sRevokeRequiresSecretName(t *testing.T) {
 	}
 }
 
+// TestK8sTLSTrust proves the ca=/insecure= plumbing actually drives the TLS
+// handshake when no client is injected (the real-apiserver path). The httptest
+// server serves a self-signed cert, so a default client rejects it; pinning its CA
+// or setting insecure=1 must make the same request succeed.
+func TestK8sTLSTrust(t *testing.T) {
+	v := vault.New()
+	defer v.Purge()
+	ctx := context.Background()
+	const ns, sa, tok = "labns", "labsa", "eyJtls0123"
+
+	mux := http.NewServeMux()
+	mux.HandleFunc("/api/v1/namespaces/"+ns+"/serviceaccounts/", func(w http.ResponseWriter, r *http.Request) {
+		json.NewEncoder(w).Encode(map[string]any{"metadata": map[string]any{"name": sa}})
+	})
+	srv := httptest.NewTLSServer(mux)
+	defer srv.Close()
+	host := strings.TrimPrefix(srv.URL, "https://")
+
+	caPEM := pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: srv.Certificate().Raw})
+	caB64 := base64.StdEncoding.EncodeToString(caPEM)
+
+	verifyWith := func(blob string) error {
+		k := &K8s{} // NO injected client — exercises clientFor's real TLS path
+		h := v.Store([]byte(blob))
+		return k.Verify(ctx, h, v)
+	}
+
+	base := "k8s://" + host + "/?ns=" + ns + "&sa=" + sa + "&token=" + tok
+
+	if err := verifyWith(base); err == nil {
+		t.Error("untrusted self-signed cert should fail without ca=/insecure=")
+	}
+	if err := verifyWith(base + "&insecure=1"); err != nil {
+		t.Errorf("insecure=1 should skip verification: %v", err)
+	}
+	if err := verifyWith(base + "&ca=" + url.QueryEscape(caB64)); err != nil {
+		t.Errorf("pinned ca= should trust the server cert: %v", err)
+	}
+
+	// Blob round-trip: ca=/insecure= survive build()->parse().
+	cr := k8sCred{base: "https://" + host, ns: ns, sa: sa, token: tok, ca: caB64, insecure: true}
+	got, err := parseK8sCred(v, v.Store([]byte(cr.build())))
+	if err != nil {
+		t.Fatalf("round-trip parse: %v", err)
+	}
+	if got.ca != caB64 || !got.insecure {
+		t.Errorf("ca/insecure did not round-trip: ca-match=%v insecure=%v", got.ca == caB64, got.insecure)
+	}
+}
+
 func TestK8sRejectsBadSecret(t *testing.T) {
 	v := vault.New()
 	defer v.Purge()

internal/rotate/proofs.go

@@ -54,6 +54,8 @@ var proofLevels = map[string]ProofLevel{
 	"wireguard": ProofLiveVM, // real wg          (lab-provision-wg.sh)
 	"gitea":     ProofLiveVM, // real Gitea       (lab-provision-gitea.sh)
 	"appsecret": ProofLiveVM, // real local files (lab-provision-appsec.sh)
+	"mongo":     ProofLiveVM, // real mongod 8.0  (lab-provision-mongo.sh)
+	"k8s":       ProofLiveVM, // real k3s v1.35   (lab-provision-k8s.sh)
 
 	// proven only against an emulator / mock:
 	"aws":        ProofMockOnly, // moto mock AWS in VM; real AWS never hit
@@ -63,12 +65,10 @@ var proofLevels = map[string]ProofLevel{
 	"cloudflare": ProofMockOnly, // httptest emulator; no self-host
 	"ghactions":  ProofMockOnly, // httptest emulator; no self-host
 	"gitlab":     ProofMockOnly, // httptest emulator; GitLab CE too heavy for the VM
-	"mongo":      ProofMockOnly, // fake-mongosh stub; real-MongoDB VM POC not yet run
 	"npm":        ProofMockOnly, // httptest emulator; real registry never hit
 	"gcp":        ProofMockOnly, // httptest emulator (real RS256 JWT signing); no self-host
 	"twilio":     ProofMockOnly, // httptest emulator; no self-host
 	"flyio":      ProofMockOnly, // httptest GraphQL emulator; no self-host
-	"k8s":        ProofMockOnly, // httptest apiserver emulator; real k3s POC not yet run
 }
 
 // ProofLevelOf returns the recorded proof level for a driver name. Unknown or