awslabs / awslabs/cli-agent-orchestrator

[Feat] Harden agent execution isolation beyond pod separation (deliberately not in 3.0)

Open
#784 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
1.3k
Forks
267
Avg merge
1d 23h
Merged PRs (30d)
70

Description

Part of #777. **Follow-on execution hardening.** This records the stronger runtime options and the conditions under which an operator needs them.

## The decision

3.0 establishes application tenant/owner isolation and separately managed execution workloads (#774/#778/#745). A mandatory sandbox runtime is not part of that default. This issue adds stronger containment where the executed code and reachable privileges require it.

The baseline is more than "different pods": approved workload templates, storage/credential ownership, restricted placement permissions and network boundaries already belong to #745. A pod boundary does not automatically make shared mounts, credentials or network access private.

## How this relates to the foundation

#778 defines a tenant as an organisation and supports a multi-organisation data model. A one-organisation reference deployment is a starting profile, not a security guarantee and not a new definition of tenancy.

Application isolation is the coordinated work in #774/#778/#779, including both tenant and private-owner access, direct addressing, discovery, files and execution context. It is not a one-query fix: a tenant-only discovery filter would still permit access between private owners within that tenant. This issue must build on the complete application boundary rather than substitute sandboxing for it.

## What would make this required

**A requirement to contain code beyond the baseline workload boundary.** Mutually untrusting customers are an important case, but not the only one. Third-party repositories, dependencies and manipulated agent actions can introduce untrusted execution inside a single organisation too (#785). Assess code provenance, data sensitivity, accessible credentials and host/network privileges; colleague status and tenant count are not substitutes for that assessment.

[Kubernetes multi-tenancy guidance](https://kubernetes.io/docs/concepts/security/multi-tenancy/) describes a spectrum of isolation and recommends additional containment for untrusted workloads. [GKE Sandbox](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/sandbox-pods) explains its userspace-kernel boundary and its limits. Select an appropriate runtime or separately isolated deployment before accepting workloads whose required containment exceeds the default; do not promise that all single-organisation deployments are safe without it.

## What this would involve

- **An appropriate sandbox/VM execution boundary** — for example gVisor, Kata, or a Firecracker-based integration. `RuntimeClass` selects an installed Kubernetes runtime handler; naming Firecracker is not itself a working handler or deployment integration.
- **Egress control**, because sandboxing explicitly does not help here. GKE's documentation is clear that a sandbox does not protect external dependencies. Default-deny egress with an allowlist.
- **Protecting node/host credentials**, including metadata-service exposure, while deliberately allowing only approved workload identity paths needed by the task. Preserve #745's credential/permission boundary rather than deferring it here.
- **Retaining #774/#745's restricted task credentials and #779's revocation behavior.** #783 adds independently managed service accounts; it is not the prerequisite for scoped agent credentials.
- **Pod Security Standards `Restricted` for agent workloads.** Baseline workload resource bounds belong to #745; aggregate tenant quota policy belongs to #781 and must account for the actual namespace topology.

## Scheduling is not sufficient by itself

Scheduling selectors alone are not a secure admission boundary if tenants can choose or spoof the labels/tolerations that select protected nodes. [GKE's guidance](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/multitenancy-overview) explains that limitation. Controlled node separation can still reduce resource contention and some exposure; it must be combined with admission/access controls and is not complete hostile-code containment.

## Acceptance criteria

*(For when this is picked up — not for 3.0.)*

- [ ] The selected sandbox/VM boundary is installed and enforced for the covered workloads; unsupported configurations fail explicitly rather than silently falling back to ordinary execution.
- [ ] Egress is default-deny with an explicit allowlist.
- [ ] Node/host credentials are unreachable from the agent; any permitted workload-identity endpoint is documented and bounded to its approved task authority.
- [ ] Credentials available to an agent are short-lived and task-scoped.
- [ ] Pod Security Standards `Restricted` is enforced for the covered agent workloads.
- [ ] Documentation states plainly which threats this does and does not address.
- [ ] The assessment covers untrusted-code scenarios within one organisation as well as mutually untrusting customers; neither organisation count nor pod placement alone is accepted as proof.

## Evidence

| Claim | Where |
| --- | --- |
| Discovery is filtered by session + group prefix, not identity | `clients/database.py:1598` (`list_siblings_by_group_prefix`); prefix filter at `:1664` |
| `cross_session=true` lifts the session filter | `mcp_server/server.py:1117-1135` |
| Group is not a security boundary | `mcp_server/server.py:1156` |
| Workload trust and isolation form a spectrum | Kubernetes multi-tenancy guidance |
| gVisor re-implements and services syscalls on behalf of the host kernel; sandbox does not cover dependencies | GKE Sandbox documentation |
| Sandboxed runtimes via `RuntimeClass`; VM boundary option | Kubernetes multi-tenancy documentation |
| Labels/tolerations alone cannot enforce policy against callers able to spoof them | GKE multi-tenancy guidance |

Contributor guide

Open the contributing guide

Research direction

Start by reading the evidence in clients/database.py:1598 and :1664 and mcp_server/server.py:1117-1135 and :1156, then review the linked #745, #774, #778, #779, #781 and #783 foundations. Compare the Kubernetes, GKE Sandbox, RuntimeClass and egress guidance against the acceptance criteria. Done means selecting and enforcing a supported containment boundary with explicit failure, bounded credentials, default-deny egress, Restricted Pod Security Standards and documented limits.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, python
Domain
infrastructure, security
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.