Attaching device-plugin networks after container start silently kills published ports, breaking cluster kernels
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 15h 13m
- Merged PRs (30d)
- 368
Description
## Symptom
On a production cluster, every multi-node (cluster_size >= 2) session involving a node with an extra device-plugin network (a macvlan RDMA network in our case) failed identically:
- the kernel container starts fine (runner up, sshd/ttyd started within ~1.3s per container logs)
- the agent's create flow sits at "waiting for kernel service initialization" for the full 60s polling window and never reaches the kernel's repl port
- the create RPC times out, the manager retries and destroys the session — kernels end as terminated-unknown-container / already-terminated, and from the user's perspective the deployment is stuck in Creating and then dies (~2 min cycle)
Single-node sessions on the same node are unaffected.
## Root cause
DockerAgent._attach_additional_networks (src/ai/backend/agent/docker/agent.py, ~L1057) connects the networks returned by AbstractComputePlugin.get_docker_networks() _after_ the container has been created and started (call site ~L1355-1360).
Docker tears down and re-programs the container's port mappings on "network connect", and for a macvlan network the published ports never come back: the docker-proxy listeners for the container's published ports (including the intrinsic repl ports 2000/2001) die at that moment. The agent then cannot reach the kernel runner at the published repl address, even though the container itself is perfectly healthy.
## Minimal reproduction (plain Docker, no Backend.AI needed)
\```Java
$ docker network create -d overlay --attachable testov # on a swarm manager
$ docker network create -d macvlan -o parent= testmv
$ docker run -d --rm --name repro --network testov -p 127.0.0.1::7681 sleep 120
$ ss -tlnp | grep $(docker port repro 7681 | cut -d: -f2) # -> docker-proxy LISTENING
$ docker network connect testmv repro
$ ss -tlnp | grep $(docker port repro 7681 | cut -d: -f2) # -> GONE (no listener)
\
```
We verified the inverse on the affected cluster: removing the plugin-attached macvlan network made a previously-always-failing multi-node pair reach RUNNING in 10 seconds.
## Why this is easy to misdiagnose
The container is healthy and its logs show a clean start, the agent logs no error (the repl polling retries ZMQError silently until the deadline), and the session's terminal states point at lifecycle bookkeeping rather than networking.
## Suggested fixes
# Attach the device-plugin networks at container creation time via NetworkingConfig/EndpointsConfig instead of a post-start "network connect", so port publishing is programmed once with the final network set; or
# If post-start attach must remain, re-verify the published ports after the attach (and fail fast with a clear error when the publish is gone), so the failure surfaces as an actionable message instead of a silent 60s timeout; and
# Log each additional-network attach at INFO with the network driver — a macvlan attach is currently invisible in the agent log.
JIRA Issue: BA-7008
Contributor guide
Research direction
Start in src/ai/backend/agent/docker/agent.py at DockerAgent._attach_additional_networks and its call site around lines 1355-1360. Reproduce the network-connect behavior with the Docker commands in the issue, then trace how NetworkingConfig/EndpointsConfig and published repl ports are handled. Done means additional networks no longer remove published ports, or the agent fails fast with a clear error and logs the network driver.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, python
- Domain
- backend, distributed-systems, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100