diff --git a/docs/FINDINGS-2026-06-16.md b/docs/FINDINGS-2026-06-16.md index a1aafa1..f53678c 100644 --- a/docs/FINDINGS-2026-06-16.md +++ b/docs/FINDINGS-2026-06-16.md @@ -290,14 +290,30 @@ just an `HTTPClient`/`Bin`), validated against protocol-enforcing emulators: for the controller to populate `.data.token`, GETs the SA to verify, then DELETEs the old Secret (which invalidates the old token). -**All 6 are MOCK-ONLY** — honestly, because they were proven only against emulators. mongo, -npm and k8s are LIVE-VM *candidates* (real MongoDB / a throwaway npm registry / a k3s cluster -would promote them with zero driver-code change), but per the standing rule a driver is only -LIVE-VM after the *real target* validates the cutover, so they stay MOCK-ONLY until those VM -POCs run. Each emulator enforces credential validity and the tests assert `Verify(old)` fails +Each emulator enforces credential validity and the tests assert `Verify(old)` fails after revoke, plus a leak canary (no token/secret/`PRIVATE KEY` in Identity/Source/Location). -Full suite after batch 2: **103 green**, `-race` clean on rotate/sink/vault. +### Live-VM POCs run (2026-06-18/19) → mongo + k8s promoted to LIVE-VM + +- **mongo → LIVE-VM** (`lab-provision-mongo.sh`): real `mongod` 8.0 from the official apt + repo; `labapp` (admin, `userAdminAnyDatabase`) rotates its own password; old password + rejected, new works. *Surfaced by the real target:* real `mongosh` echoes a `labdb> ` prompt + to stdout on a pipe and `db.auth` returns an object not a bool in 2.8 — the fake-mongosh stub + did neither, so the assertion had to filter output to the `AUTH-OK`/`AUTH-FAIL` token. +- **k8s → LIVE-VM** (`lab-provision-k8s.sh`): real **k3s v1.35** kube-apiserver + token + controller; `labsa` (RBAC: create/delete/get Secrets + get its SA) rotates its legacy + token Secret; old token → `401`, new → `200`. *Surfaced by the real target:* (a) the + apiserver serves a self-signed cert, so the driver needed CA-pinned TLS — added `ca=` + (base64 PEM) + an `insecure=1` lab-only escape hatch to the blob; (b) the apiserver caches + *successful* auths ~10s (`cachedTokenAuthenticator`), so the deleted token kept working until + the cache expired (~7s) — the POC polls past it. Both are real-deployment config, not + test-awareness. +- **npm stays MOCK-ONLY** (decided, not pending): `registry.npmjs.org` isn't self-hostable and + real `POST /-/npm/v1/tokens` requires the **account password in the body** (the driver sends + only Bearer old-token). Documented as a contract gap in `docs/ROTATION-PROOFS.md` rather than + faking a LIVE-VM. gcp/twilio/flyio remain MOCK-ONLY (no self-host). + +Full suite after batch 2 + POCs: **104 green**, `-race` clean on rotate/sink/vault. ## Summary diff --git a/docs/ROTATION-PROOFS.md b/docs/ROTATION-PROOFS.md index 903e77a..6c6ca38 100644 --- a/docs/ROTATION-PROOFS.md +++ b/docs/ROTATION-PROOFS.md @@ -41,6 +41,17 @@ stops a mock-only driver from ever being mistaken for a live one. | `wireguard` | real `wg` | `lab-provision-wg.sh` | | `gitea` | real Gitea | `lab-provision-gitea.sh` | | `appsecret` | real local files | `lab-provision-appsec.sh` | +| `mongo` | real `mongod` 8.0 | `lab-provision-mongo.sh` | +| `k8s` | real k3s `v1.35` kube-apiserver | `lab-provision-k8s.sh` | + +> **k8s live-POC note (auth-cache timing):** the real kube-apiserver caches *successful* +> token authentications for ~10s (`cachedTokenAuthenticator`). After `RevokeOld` deletes the +> legacy token Secret, the old token keeps authenticating until that cache entry expires — so +> the POC polls the old token for up to 30s and asserts it flips to `401` (it does, ~7s). The +> `httptest` emulator invalidates instantly, so this delay only surfaced against real k3s — +> exactly what a live POC is for. The driver needed real-CA TLS to talk to the apiserver, so +> the blob gained `ca=` (base64 PEM, pinned as the only trusted root) and an `insecure=1` +> lab-only escape hatch; neither is test-awareness — a real deployment configures the same CA. ### MOCK-ONLY — proven only against an emulator / mock @@ -53,12 +64,10 @@ stops a mock-only driver from ever being mistaken for a live one. | `cloudflare` | `httptest` emulator | no self-host | | `ghactions` | `httptest` emulator (holds the libsodium box keypair, decrypts the sealed PUT) | no self-host | | `gitlab` | `httptest` emulator | GitLab CE too heavy for the VM | -| `mongo` | fake-`mongosh` shell stub (auths against a state file) | real-MongoDB VM POC not yet run | -| `npm` | `httptest` registry emulator | real registry never hit | +| `npm` | `httptest` registry emulator | real registry never hit (see caveat below) | | `gcp` | `httptest` IAM emulator (verifies the driver's RS256 JWT against the key's public key) | no self-host | | `twilio` | `httptest` emulator (Basic-auth Keys resource) | no self-host | | `flyio` | `httptest` GraphQL emulator | no self-host | -| `k8s` | `httptest` apiserver emulator (token-Secret create/delete) | real k3s POC not yet run | ## What the MOCK-ONLY emulators actually prove @@ -73,7 +82,7 @@ protocol so the test exercises the driver's true cutover logic: on `PUT` **decrypts the sealed value** and stores the plaintext; the test asserts the decrypted value equals the new value the driver put in the rebuilt blob — proving the sealed-box encryption is correct, not simulated. -- **npm / twilio / k8s** — each emulator enforces credential validity (a token/key +- **npm / twilio** — each emulator enforces credential validity (a token/key authenticates only while it exists) and the test asserts `Verify(old)` fails after `RevokeOld` — proving the create→verify→revoke cutover really happened, old credential dead, new one alive. @@ -85,10 +94,17 @@ protocol so the test exercises the driver's true cutover logic: - **flyio** — a managing token mints/deletes the deploy token over GraphQL; the deploy token authenticates the app query only while it exists, so the test proves the managing-token → deploy-token rotation and that the revoked deploy token stops working. -- **mongo** — the fake `mongosh` authenticates `db.auth` against a password held in a state - file and only `changeUserPassword` updates it; the test proves the in-place cutover (old - password rejected after rotation). No password ever reaches `mongosh`'s argv. -Promoting any of these to LIVE-VM only requires standing up the real service (a self-hosted -GitLab CE / a real Cloudflare token / a throwaway GitHub repo / a real MongoDB / a real npm -registry / a k3s cluster) and re-running the same driver against it — no driver code changes. +> **npm caveat (why it is not just "pending a VM run"):** the public `registry.npmjs.org` +> is not self-hostable, and the classic `POST /-/npm/v1/tokens` create-token call on real +> npm **requires the account password in the request body** (it is the non-interactive form +> of `npm token create`, which prompts for your password). The driver authenticates the +> create with the *old token* as Bearer only, which the emulator accepts but real npmjs does +> not. So `npm` stays MOCK-ONLY pending either (a) a self-hosted registry that implements the +> token API faithfully, or (b) adding password-in-body support to the create call. This is +> exactly the kind of contract gap a live-POC attempt is meant to surface. + +Promoting the rest to LIVE-VM only requires standing up the real service (a self-hosted +GitLab CE / a real Cloudflare token / a throwaway GitHub repo) and re-running the same +driver against it — no driver code changes. (`k8s` and `mongo` have already made this jump; +see the LIVE-VM table above.) diff --git a/internal/rotate/k8s.go b/internal/rotate/k8s.go index a49c684..f815fbc 100644 --- a/internal/rotate/k8s.go +++ b/internal/rotate/k8s.go @@ -3,6 +3,8 @@ package rotate import ( "bytes" "context" + "crypto/tls" + "crypto/x509" "encoding/base64" "encoding/json" "fmt" @@ -32,6 +34,10 @@ import ( // - sa= the ServiceAccount name (the new Secret is annotated to it). // - secret= the token Secret's name — required to DELETE the old token. // - token= the bearer token value (rotated). +// - ca= OPTIONAL base64(std) of the cluster CA PEM bundle. A real apiserver +// serves a self-signed cert, so the driver pins this CA as the only +// trusted root for the TLS handshake. Omit for a loopback emulator. +// - insecure=1 OPTIONAL escape hatch: skip TLS verification entirely (lab only). // // AUTH: every call authenticates with a token Bearer header against the apiserver. The // rotating SA needs RBAC to create/delete Secrets in its namespace. The new token is @@ -53,20 +59,44 @@ func (k *K8s) Name() string { return "k8s" } // Detect claims credentials tagged Source == "k8s". func (k *K8s) Detect(c discover.Credential) bool { return c.Source == "k8s" } -func (k *K8s) client() *http.Client { +// clientFor returns the HTTP client for a given credential. An injected client +// (tests / explicit CA config) always wins. Otherwise it builds a 15s client whose +// TLS trust is pinned to the credential's ca= bundle (a real apiserver's self-signed +// cert), or with verification disabled when insecure=1 is set (lab only). +func (k *K8s) clientFor(cr k8sCred) (*http.Client, error) { if k.HTTPClient != nil { - return k.HTTPClient + return k.HTTPClient, nil } - return &http.Client{Timeout: 15 * time.Second} + tlsCfg := &tls.Config{MinVersion: tls.VersionTLS12} + switch { + case cr.insecure: + tlsCfg.InsecureSkipVerify = true + case cr.ca != "": + pem, err := base64.StdEncoding.DecodeString(cr.ca) + if err != nil { + return nil, fmt.Errorf("k8s: decode ca bundle: %w", err) + } + pool := x509.NewCertPool() + if !pool.AppendCertsFromPEM(pem) { + return nil, fmt.Errorf("k8s: ca bundle contained no usable certificates") + } + tlsCfg.RootCAs = pool + } + return &http.Client{ + Timeout: 15 * time.Second, + Transport: &http.Transport{TLSClientConfig: tlsCfg}, + }, nil } // k8sCred is the parsed credential blob. None of its fields are ever logged. type k8sCred struct { - base string // scheme://host:port - ns string - sa string - secret string // token Secret name - token string + base string // scheme://host:port + ns string + sa string + secret string // token Secret name + token string + ca string // base64(std) cluster CA PEM bundle (optional) + insecure bool // skip TLS verification (lab only) } func parseK8sCred(v *vault.Vault, h *vault.Handle) (k8sCred, error) { @@ -89,7 +119,7 @@ func parseK8sCred(v *vault.Vault, h *vault.Handle) (k8sCred, error) { return k8sCred{}, fmt.Errorf("k8s: secret has no apiserver host") } q := u.Query() - c := k8sCred{base: scheme + "://" + u.Host, ns: q.Get("ns"), sa: q.Get("sa"), secret: q.Get("secret"), token: q.Get("token")} + c := k8sCred{base: scheme + "://" + u.Host, ns: q.Get("ns"), sa: q.Get("sa"), secret: q.Get("secret"), token: q.Get("token"), ca: q.Get("ca"), insecure: q.Get("insecure") == "1"} if c.ns == "" || c.sa == "" { return k8sCred{}, fmt.Errorf("k8s: secret missing namespace or serviceaccount") } @@ -110,6 +140,12 @@ func (c k8sCred) build() string { q.Set("secret", c.secret) } q.Set("token", c.token) + if c.ca != "" { + q.Set("ca", c.ca) + } + if c.insecure { + q.Set("insecure", "1") + } u.RawQuery = q.Encode() return u.String() } @@ -132,8 +168,9 @@ type k8sSecretObj struct { Data map[string]string `json:"data,omitempty"` } -// do issues an authenticated apiserver request and returns the body for 2xx responses. -func (k *K8s) do(ctx context.Context, method, urlStr, token string, body []byte) ([]byte, int, error) { +// do issues an authenticated apiserver request and returns the body. The credential +// carries the TLS trust config (ca=/insecure=) used to build the client. +func (k *K8s) do(ctx context.Context, cr k8sCred, method, urlStr string, body []byte) ([]byte, int, error) { var r io.Reader if body != nil { r = bytes.NewReader(body) @@ -142,11 +179,15 @@ func (k *K8s) do(ctx context.Context, method, urlStr, token string, body []byte) if err != nil { return nil, 0, err } - req.Header.Set("Authorization", "Bearer "+token) + req.Header.Set("Authorization", "Bearer "+cr.token) if body != nil { req.Header.Set("Content-Type", "application/json") } - resp, err := k.client().Do(req) + cl, err := k.clientFor(cr) + if err != nil { + return nil, 0, err + } + resp, err := cl.Do(req) if err != nil { return nil, 0, err } @@ -181,7 +222,7 @@ func (k *K8s) Rotate(ctx context.Context, c discover.Credential, v *vault.Vault) obj.Metadata.Annotations = map[string]string{"kubernetes.io/service-account.name": cr.sa} body, _ := json.Marshal(obj) - _, code, err := k.do(ctx, http.MethodPost, cr.secretsURL(""), cr.token, body) + _, code, err := k.do(ctx, cr, http.MethodPost, cr.secretsURL(""), body) if err != nil { return nil, fmt.Errorf("k8s: create token secret: %w", err) } @@ -203,7 +244,7 @@ func (k *K8s) Rotate(ctx context.Context, c discover.Credential, v *vault.Vault) // awaitToken polls the new Secret until its .data.token is present, decoding it. func (k *K8s) awaitToken(ctx context.Context, cr k8sCred, name string) (string, error) { for attempt := 0; attempt < 10; attempt++ { - rb, code, err := k.do(ctx, http.MethodGet, cr.secretsURL(name), cr.token, nil) + rb, code, err := k.do(ctx, cr, http.MethodGet, cr.secretsURL(name), nil) if err != nil { return "", fmt.Errorf("k8s: read token secret: %w", err) } @@ -236,7 +277,7 @@ func (k *K8s) Verify(ctx context.Context, newSecret *vault.Handle, v *vault.Vaul return err } saURL := cr.base + "/api/v1/namespaces/" + url.PathEscape(cr.ns) + "/serviceaccounts/" + url.PathEscape(cr.sa) - _, code, err := k.do(ctx, http.MethodGet, saURL, cr.token, nil) + _, code, err := k.do(ctx, cr, http.MethodGet, saURL, nil) if err != nil { return fmt.Errorf("k8s verify: %w", err) } @@ -257,7 +298,7 @@ func (k *K8s) RevokeOld(ctx context.Context, c discover.Credential, v *vault.Vau if cr.secret == "" { return fmt.Errorf("k8s revoke: old token has no recorded secret name — delete it manually (guided)") } - _, code, err := k.do(ctx, http.MethodDelete, cr.secretsURL(cr.secret), cr.token, nil) + _, code, err := k.do(ctx, cr, http.MethodDelete, cr.secretsURL(cr.secret), nil) if err != nil { return fmt.Errorf("k8s revoke: %w", err) } diff --git a/internal/rotate/k8s_test.go b/internal/rotate/k8s_test.go index 90abbd8..7d5cf02 100644 --- a/internal/rotate/k8s_test.go +++ b/internal/rotate/k8s_test.go @@ -6,8 +6,10 @@ import ( "encoding/base64" "encoding/hex" "encoding/json" + "encoding/pem" "net/http" "net/http/httptest" + "net/url" "strings" "sync" "testing" @@ -218,6 +220,56 @@ func TestK8sRevokeRequiresSecretName(t *testing.T) { } } +// TestK8sTLSTrust proves the ca=/insecure= plumbing actually drives the TLS +// handshake when no client is injected (the real-apiserver path). The httptest +// server serves a self-signed cert, so a default client rejects it; pinning its CA +// or setting insecure=1 must make the same request succeed. +func TestK8sTLSTrust(t *testing.T) { + v := vault.New() + defer v.Purge() + ctx := context.Background() + const ns, sa, tok = "labns", "labsa", "eyJtls0123" + + mux := http.NewServeMux() + mux.HandleFunc("/api/v1/namespaces/"+ns+"/serviceaccounts/", func(w http.ResponseWriter, r *http.Request) { + json.NewEncoder(w).Encode(map[string]any{"metadata": map[string]any{"name": sa}}) + }) + srv := httptest.NewTLSServer(mux) + defer srv.Close() + host := strings.TrimPrefix(srv.URL, "https://") + + caPEM := pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: srv.Certificate().Raw}) + caB64 := base64.StdEncoding.EncodeToString(caPEM) + + verifyWith := func(blob string) error { + k := &K8s{} // NO injected client — exercises clientFor's real TLS path + h := v.Store([]byte(blob)) + return k.Verify(ctx, h, v) + } + + base := "k8s://" + host + "/?ns=" + ns + "&sa=" + sa + "&token=" + tok + + if err := verifyWith(base); err == nil { + t.Error("untrusted self-signed cert should fail without ca=/insecure=") + } + if err := verifyWith(base + "&insecure=1"); err != nil { + t.Errorf("insecure=1 should skip verification: %v", err) + } + if err := verifyWith(base + "&ca=" + url.QueryEscape(caB64)); err != nil { + t.Errorf("pinned ca= should trust the server cert: %v", err) + } + + // Blob round-trip: ca=/insecure= survive build()->parse(). + cr := k8sCred{base: "https://" + host, ns: ns, sa: sa, token: tok, ca: caB64, insecure: true} + got, err := parseK8sCred(v, v.Store([]byte(cr.build()))) + if err != nil { + t.Fatalf("round-trip parse: %v", err) + } + if got.ca != caB64 || !got.insecure { + t.Errorf("ca/insecure did not round-trip: ca-match=%v insecure=%v", got.ca == caB64, got.insecure) + } +} + func TestK8sRejectsBadSecret(t *testing.T) { v := vault.New() defer v.Purge() diff --git a/internal/rotate/proofs.go b/internal/rotate/proofs.go index e386544..8785a86 100644 --- a/internal/rotate/proofs.go +++ b/internal/rotate/proofs.go @@ -54,6 +54,8 @@ var proofLevels = map[string]ProofLevel{ "wireguard": ProofLiveVM, // real wg (lab-provision-wg.sh) "gitea": ProofLiveVM, // real Gitea (lab-provision-gitea.sh) "appsecret": ProofLiveVM, // real local files (lab-provision-appsec.sh) + "mongo": ProofLiveVM, // real mongod 8.0 (lab-provision-mongo.sh) + "k8s": ProofLiveVM, // real k3s v1.35 (lab-provision-k8s.sh) // proven only against an emulator / mock: "aws": ProofMockOnly, // moto mock AWS in VM; real AWS never hit @@ -63,12 +65,10 @@ var proofLevels = map[string]ProofLevel{ "cloudflare": ProofMockOnly, // httptest emulator; no self-host "ghactions": ProofMockOnly, // httptest emulator; no self-host "gitlab": ProofMockOnly, // httptest emulator; GitLab CE too heavy for the VM - "mongo": ProofMockOnly, // fake-mongosh stub; real-MongoDB VM POC not yet run "npm": ProofMockOnly, // httptest emulator; real registry never hit "gcp": ProofMockOnly, // httptest emulator (real RS256 JWT signing); no self-host "twilio": ProofMockOnly, // httptest emulator; no self-host "flyio": ProofMockOnly, // httptest GraphQL emulator; no self-host - "k8s": ProofMockOnly, // httptest apiserver emulator; real k3s POC not yet run } // ProofLevelOf returns the recorded proof level for a driver name. Unknown or