agent-substrate / agent-substrate/substrate
Add trustBundle as a SystemInfo volume data source
- 主要語言
- Go
- 星號
- 1.8k
- 分支
- 316
- 平均合併
- 2 天 43 分鐘
- 30 天內合併 PR
- 287
描述
## Summary
Add a `trustBundle` data source to SystemInfo volumes (#802) that projects the trust anchors of a named trust bundle to a PEM file in the volume — inspired by the [Kubernetes clusterTrustBundle projected volume source](https://kubernetes.io/docs/concepts/storage/projected-volumes/#clustertrustbundle), but deliberately source-neutral: the substrate API names a bundle, and how that bundle is fetched is a deployment concern behind ateapi. Initially the only supported bundle is the well-known `egress-mitm.ate.dev`, resolved from a Kubernetes ClusterTrustBundle.
## Motivation
#871 lands MITM'd egress TLS: Envoy terminates actor egress with short-lived per-SNI leaves minted under the egress-mitm-ca-pool CA. It explicitly leaves one problem open: how to make actors trust that CA. The plan (per the egress design on #823) is a trust bundle maintained by atecontroller from the CA pool, published as a Kubernetes ClusterTrustBundle. This issue introduces the mechanism for getting those trust anchors onto each actor's filesystem, correct across suspend/resume and snapshot seeding.
This is precisely what SystemInfo volumes exist for (#802): per-actor files generated by the platform on every Run/Restore, never captured into checkpointed state.
Note the k8s ClusterTrustBundle is a distribution channel here, not the source of truth (the CA pool is), and substrate APIs should not take a hard dependency on Kubernetes APIs. The data source is therefore named `trustBundle`, not `clusterTrustBundle`: templates reference a bundle by name, and resolving that name to a backing store is substrate's job. Initially we will just support the well known `egress-mitm.ate.dev`, fetched from the ClusterTrustBundle of the same name. Eventually we will introduce a configurable backend registry so users can specify the trustBundles that are available to their actors and where they can be fetched from.
## Proposed API
```yaml
spec:
volumes:
- name: trust
systemInfo:
dataSources:
- trustBundle:
name: egress-mitm.ate.dev
path: egress-ca.pem
containers:
- name: main
volumeMounts:
- name: trust
mountPath: /run/substrate/certs # e.g. the actor reads /run/substrate/certs/egress-ca.pem
```
## Semantics
- name is resolved by ateapi at actor start against the supported-bundle allowlist. Initially the allowlist contains only egress-mitm.ate.dev, which ateapi resolves through an informer-backed lister to the Kubernetes ClusterTrustBundle of the same name, sanitized the way kubelet does for projections: only CERTIFICATE PEM blocks are kept, deduplicated, headers stripped.
- The allowlist lives in ateapi, not the CRD schema: an unknown bundle name is schema-valid but fails actor start with an error naming it. That keeps the future backend registry an ateapi change, not an API change.
- The wire spec carries only {path, pem_bundle} resolved bytes. atelet writes them to a stable path via temp-file + rename and never talks to any bundle backend.
- Fail-closed with a clear error: starting the actor fails, naming the bundle, if it is not in the allowlist, or its backend is unavailable in this deployment, or the resolved bundle is missing, empty, or contains no certificates.
- Re-resolved on every Run/Restore, like all SystemInfo contents. A resumed actor always gets the bundle's current state, regardless of what snapshot it booted from.
- Kubernetes-backend specifics (implementation detail of the initial backend, not the API): certificates.k8s.io/v1beta1 is feature-gated as of Kubernetes v1.36 (our kind clusters enable it). ateapi probes for the API at startup and degrades gracefully when absent — templates referencing a bundle then fail actor start with a clear error rather than ateapi hanging at boot. ateapi needs RBAC to read clustertrustbundles.certificates.k8s.io.
## Update & rotation semantics
- Single file, replaced by rename at a stable path — not kubelet's AtomicWriter. The k8s clusterTrustBundle projection rides kubelet's timestamped-dir + ..data symlink machinery, but #803 removed our vendored copy of it for cause: the guest records the resolved timestamped paths at suspend, and since SystemInfo volumes are rebuilt from scratch at resume (possibly on another node), those wall-clock names can never be reproduced — the restore hard-fails under virtiofsd's find-paths migration. A one-file bundle in a fixed-shape volume doesn't need AtomicWriter's whole-set atomicity anyway (that exists for cert/key pairs and volumes whose file set changes at runtime); temp-file + rename in the same directory is already atomic for readers and keeps every recorded path stable.
- Correctness never depends on propagation latency. The invariant belongs to the bundle publisher (the atecontroller reconciler maintaining egress-mitm.ate.dev from the CA pool): rotate with overlap — the new CA enters the bundle before any leaf is minted under it, and the old CA stays until its last leaf expires plus the propagation window. The reader contract matches podCertificate projections: the platform rewrites the file, apps re-read to pick it up. Runtimes that load trust once at process start won't see live updates at all, so the overlap window must comfortably dominate the reconcile interval.
- No bundle GC, no bundle history in snapshots. Rename-in-place leaves the old inode to the kernel's refcounting while the actor runs; suspend tears down virtiofsd and releases the rest; resume re-binds held fds by the stable path to the freshly generated bundle. One case is unmigratable by construction: an fd held across a live update, then suspended before re-opening — the inode is unlinked, so find-paths has no path to record, and retaining old bundles (on disk or in snapshots) cannot help, which is also why bundle history stays out of snapshots. Under virtiofsd's default --migration-on-error=abort that one fd fails the whole restore; guest-error would degrade it to EIO on that fd, healed by re-open. Since #846 the micro-VM runtime serves everything over one unified share, so the flag is share-wide — flipping it needs its own discussion, likely alongside PR 2.
## Tasks
- [ ] PR 1 — trustBundle data source. CRD member + CEL validation, the
{path, pem_bundle} wire member, the ateapi allowlist (initially only
egress-mitm.ate.dev) with resolution through an informer-backed lister
and kubelet-style PEM sanitization, atelet write via temp-file + rename
at a stable path, ateapi RBAC for
clustertrustbundles.certificates.k8s.io, and identity-suite e2e
(fixture bundle with junk/duplicate blocks, asserting the sanitized PEM
is delivered). Bundle contents refresh on every actor Run/Restore.
- [ ] PR 2 — live refresh for running actors. The backend's event
handlers forward bundle updates to affected actors: a new atelet
UpdateSystemInfoVolume RPC, ateapi fan-out from a changed bundle to
the atelets hosting actors that project it (with reconciliation for
unreachable atelets and mid-rotation migrations), and e2e rotating the
fixture bundle under a running actor on both runtimes (gVisor bind
mount and micro-VM virtio-fs must both observe the rename-in-place at
the stable path). This matches kubelet's live-update semantics for
clusterTrustBundle projections, with the write mechanism swapped per
Update & rotation semantics above.
- [ ] PR 3 — auto-injected egress trust volume. Policy layer for #871's
open item: ateapi injects a well-known SystemInfo volume projecting
egress-mitm.ate.dev into every actor (default mount under
/run/substrate/certs/), with reserved-name/mount-collision rules and
fail-closed behavior when the bundle is absent. Depends on the
atecontroller reconciler that maintains the bundle from
egress-mitm-ca-pool; merge-ordering to be coordinated on that PR.
- [ ] Later (tracked separately when concrete): configurable trust-bundle
backends. Replace the hardcoded allowlist with a registry that lets a
deployment declare which bundle names exist and where each is fetched
from (k8s ClusterTrustBundles being one backend). Deliberately
unspecified here.
## Alternatives considered
- A clusterTrustBundle data source mirroring the k8s projected volume
source 1:1. Rejected: it bakes a Kubernetes API type into substrate's
template API. Naming the source trustBundle and treating the k8s CTB as
the first backend keeps substrate APIs free of hard k8s dependencies —
non-k8s deployments fail actor start with "backend unavailable" today and
gain real backends via the registry later, with no template API change.
貢獻指南
評估
這個 Issue 還沒有評估資料。