agent-substrate / agent-substrate/substrate

Actor egress is activated only after readyz, so a workload that fetches remote content to become ready can never start

Open
#1,543 0 comments 0 reactions 0 assignees View on GitHub
area/network kind/bug
Dominant language
Go
Stars
1.8k
Forks
316
Avg merge
2d 43m
Merged PRs (30d)
287

Description

> **Disclosure:** this issue was written by an AI agent (Claude) while verifying kagent
> documentation against a live cluster, and reviewed by a human before filing.

## Summary

An Actor has no working egress until every readyz-enabled container reports ready, but a
workload that must fetch remote content in order to *become* ready therefore cannot start.
Actor networking is activated only after `readyz.WaitAll` returns, which creates a circular
dependency for any image that pulls from the network during startup.

This is the substrate half of kagent-dev/kagent#2604, which blocks Git, OCI, and S3 Agent
Plugin sources outright. Filing here because the ordering that has to change is in
`ateom-gvisor`.

## The ordering

`cmd/ateom-gvisor/main.go`, in both the Run and the Restore paths:

| Step | Run | Restore |
| --- | --- | --- |
| `deactivateActorNetworking` | :625 | :905 |
| containers created and started | :679-700 | — |
| `readyz.WaitAll` blocks for 200 from every readyz container | **:722** | **:1031** |
| `activateActorNetworking` | **:725** | **:1034** |

Preparing the egress identity and installing the redirect is not the same as allowing
traffic. Until `activateActorNetworking` runs, the redirect has no active tunnel and the
downstream connection is closed as soon as it is made, which is why callers see a connection
that opens and immediately dies rather than a refusal or a timeout.

```text
readiness requires remote content
-> fetching remote content requires egress
-> egress activation requires readiness
```

## Observed

On a kind cluster (kagent `b8d53d39`, substrate 0.0.25), a golden Actor in the
`ate-golden` atespace failed both source kinds during startup:

```text
Initialized empty Git repository in /plugins/standalone-0/.git/
fatal: unable to access 'https://github.com/...': Send failure: Broken pipe
materialize agent plugins: materialize skill "skill-creator": exit status 128
```

```text
pull ghcr.io/.../...@sha256:d497...: Get "https://ghcr.io/v2/": EOF
```

Both end in `failed to materialize Agent Plugins` and the process exits, so the container
never reports ready and networking is never activated.

It is not a cluster-wide network problem. A normal, already-golden Actor reached its model
provider fine over the same path — `atenet-egress` logged exactly one request in two hours,
`tls.sni=api.openai.com` — and an in-cluster MCP server was reachable from an Actor. There
is no egress-gateway log line at all for github.com or ghcr.io, consistent with the traffic
never leaving the sandbox.

## Impact

The failure is also hard to diagnose from the consumer side. The template reports
`Ready=False`, reason `ActorTemplatePending`, message `waiting for the ActorTemplate golden
snapshot` — identical to a healthy template that is still booting, and it names neither the
source nor the network. The real error appears only in the worker pod logs.

## Expected

A workload can reach its declared egress destinations during startup, before it is required
to report ready. Options that would resolve it:

- Activate actor networking before waiting on readyz, so startup fetches can use the
configured egress policy.
- Or introduce an explicit initialization phase with egress enabled that runs before the
readiness gate.
- Or expose an ActorTemplate option for workloads that need egress during startup.

Whichever way it goes, a startup fetch failure should surface as a distinguishable condition
rather than presenting as "still waiting for the golden snapshot".

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.