agent-substrate / agent-substrate/substrate
Select devices per-container in the ActorTemplate
- Lenguaje dominante
- Go
- Estrellas
- 1.8k
- Forks
- 316
- Merge medio
- 2 d 43 min
- PR fusionados (30 d)
- 287
Descripción
## Problem
Follow-up to #502, which landed GPU passthrough at the worker level. A pool requests N GPUs and **every** actor container gets **all** of them. The `ActorTemplate` containers have no say in the device layout, so device assignment is coupled to the worker.
Per @BenTheElder ([review](https://github.com/agent-substrate/substrate/pull/502#pullrequestreview-4859636932)), device assignment should move into the `ActorTemplate` so it is decoupled from the worker and can control which containers in an actor receive the devices.
## Use cases
Actors are pod-shaped (multiple containers, one gVisor sandbox, shared netns), so per-container resource control unlocks:
- **Accelerator container plus CPU-only sidecar.** The model or trainer container claims the accelerator (`nvidia.com/gpu`, `amd.com/gpu`, an FPGA); the logging, proxy, or metrics sidecar next to it claims none. Today every sidecar is handed the device nodes.
- **Security isolation.** An untrusted or user-supplied sidecar is denied device access as defense in depth. Today it silently receives whatever the pod holds (`/dev/nvidia*`, an SR-IOV VF), widening the attack surface that #502 otherwise works to keep narrow. The same applies to memory and CPU: a runaway sidecar can currently consume the whole pod budget and starve the container that matters.
- **Devices split across containers.** On a multi-device node, worker containers each claim a distinct GPU for data-parallel or shard-per-container work over the shared netns.
## Proposed API
Declare resources per-container using the standard Kubernetes `limits` shape. The pool reserves capacity; each container claims from it.
```yaml
kind: WorkerPool
spec:
sandboxClass: gvisor
template:
resources:
limits:
nvidia.com/gpu: "2" # pool reserves 2 GPUs on the node (unchanged; this is what schedules onto a GPU node)
containers:
- name: trainer
image: pytorch@sha256:...
resources:
limits:
nvidia.com/gpu: "2" # this container claims both
memory: 8Gi
- name: sidecar-logger
image: fluentbit@sha256:...
resources:
limits:
memory: 256Mi # capped, and no gpu injected (today it would wrongly get both)
```
One new field on `Container`:
```go
// Resources for this container, including device-plugin resources such as
// nvidia.com/gpu.
// +optional
Resources *ContainerResources `json:"resources,omitempty"`
// ContainerResources are the resource limits for one actor container.
type ContainerResources struct {
// Limits is the maximum amount of compute resources allowed, including
// device-plugin resources such as nvidia.com/gpu.
//
// +optional
Limits corev1.ResourceList `json:"limits,omitempty"`
}
```
The YAML matches vanilla Kubernetes per-container limits, and reusing `corev1.ResourceList` keeps it device-generic for free: extended resources already cover RDMA, FPGA, and others.
## Limits only
`ContainerResources` deliberately does not reuse `corev1.ResourceRequirements`, which also carries `requests` and `claims`.
A restore needs at least the footprint that was captured — micro-VM guest RAM comes back from the memory snapshot, and under gVisor the saved pages would be faulted into a smaller cgroup. A soft request cannot express "must have this much to come back at all".
Adding `requests` later is additive and non-breaking if a use case appears. The obvious one is `cpu.weight`, relative priority between sibling containers under contention.
## Scope
- Add `resources` to `ActorTemplate.Container`, as `ContainerResources` (limits only).
- CEL validation: extend the existing `sandboxClass == 'gvisor'` guard to the container level. The type already restricts this to limits, so there is nothing else to reject at admission.
- **At assignment time**, reject an actor whose container `nvidia.com/gpu` limits sum to more than the assigned pool's reservation. This cannot be CEL: the reservation lives on the `WorkerPool`, and the template selects workers by label, so one template may match several pools with different reservations.
- Plumb the per-container limits through `WorkerAssignment` to `ateom`.
- In `ateom`, set `spec.Linux.Resources` on the OCI spec from the container's limits, and partition the pod's assigned GPUs across the containers that request them, the way the device plugin does in real Kubernetes, instead of the unconditional per-container `maybeInjectGPU`. The partition must be deterministic, ordered by container.
- Update the API guide.
The cgroup machinery already exists: `ensureContainerCgroupsPath` gives every actor container its own cgroup leaf, so `cpu.max`, `memory.max` and friends are present and unset.
## Risks
- **Device identity is not stable across resume.** CDI names are pod-local, so on restore the actor may land on different physical devices with new UUIDs and PCI IDs. Injection already re-runs on the restore path, so device paths are re-resolved. Remaining care:
- an app that cached a device UUID or PCI ID before snapshot sees a different one after;
- the container-to-device partition must be deterministic so a container keeps its ordinal across resume.
- **Pool reservation stays the scheduling knob.** Keep the pool's `resources.limits.nvidia.com/gpu` as the reservation and validate that the containers' sum does not exceed it, rather than deriving the pool total from containers. Smaller change, single scheduling knob.
## Open questions
**Can per-container CPU and memory be enforced?** google/gvisor#190 suggests cgroup limits land on the sandbox rather than the container, since one sentry backs them all. Micro-VM actors have a real guest kernel, so it may work there. Worth settling before the field covers more than devices.
## Out of scope
- **DRA.** Dynamic Resource Allocation (`resource.k8s.io`, GA in 1.34) is the home for physical index pinning, topology co-allocation, and device sharing such as MIG and time-slicing. If those are needed, revisit through a DRA `ResourceClaim` rather than extending counts.
- **CPU pinning.** `cpuset.cpus` exists on the per-container cgroup, but counts cannot express it and it belongs with topology above. Under gVisor it would also pin the sentry's host threads rather than CPUs the sandboxed app sees.
- **Non-GPU device classes.** The surface allows them; the implementation lands with the first such device.
- **Snapshot and restore of live device state**, such as a live CUDA context. Tracked separately by #779.
## References
- #627: GPU support for gVisor actors, the parent this follows up.
Guía de contribución
Evaluación
Este issue todavía no se ha evaluado.