lablup / lablup/backend.ai

Attaching device-plugin networks after container start silently kills published ports, breaking cluster kernels

Open
#13,106 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

## Symptom

On a production cluster, every multi-node (cluster_size >= 2) session involving a node with an extra device-plugin network (a macvlan RDMA network in our case) failed identically:

- the kernel container starts fine (runner up, sshd/ttyd started within ~1.3s per container logs)
- the agent's create flow sits at "waiting for kernel service initialization" for the full 60s polling window and never reaches the kernel's repl port
- the create RPC times out, the manager retries and destroys the session — kernels end as terminated-unknown-container / already-terminated, and from the user's perspective the deployment is stuck in Creating and then dies (~2 min cycle)

Single-node sessions on the same node are unaffected.

## Root cause

DockerAgent._attach_additional_networks (src/ai/backend/agent/docker/agent.py, ~L1057) connects the networks returned by AbstractComputePlugin.get_docker_networks() _after_ the container has been created and started (call site ~L1355-1360).

Docker tears down and re-programs the container's port mappings on "network connect", and for a macvlan network the published ports never come back: the docker-proxy listeners for the container's published ports (including the intrinsic repl ports 2000/2001) die at that moment. The agent then cannot reach the kernel runner at the published repl address, even though the container itself is perfectly healthy.

## Minimal reproduction (plain Docker, no Backend.AI needed)

\```Java
$ docker network create -d overlay --attachable testov # on a swarm manager
$ docker network create -d macvlan -o parent= testmv
$ docker run -d --rm --name repro --network testov -p 127.0.0.1::7681 sleep 120
$ ss -tlnp | grep $(docker port repro 7681 | cut -d: -f2) # -> docker-proxy LISTENING
$ docker network connect testmv repro
$ ss -tlnp | grep $(docker port repro 7681 | cut -d: -f2) # -> GONE (no listener)
\
```

We verified the inverse on the affected cluster: removing the plugin-attached macvlan network made a previously-always-failing multi-node pair reach RUNNING in 10 seconds.

## Why this is easy to misdiagnose

The container is healthy and its logs show a clean start, the agent logs no error (the repl polling retries ZMQError silently until the deadline), and the session's terminal states point at lifecycle bookkeeping rather than networking.

## Suggested fixes

# Attach the device-plugin networks at container creation time via NetworkingConfig/EndpointsConfig instead of a post-start "network connect", so port publishing is programmed once with the final network set; or

# If post-start attach must remain, re-verify the published ports after the attach (and fail fast with a clear error when the publish is gone), so the failure surfaces as an actionable message instead of a silent 60s timeout; and

# Log each additional-network attach at INFO with the network driver — a macvlan attach is currently invisible in the agent log.

JIRA Issue: BA-7008

Contributor guide

Open the contributing guide

Research direction

Start in src/ai/backend/agent/docker/agent.py at DockerAgent._attach_additional_networks and its call site around lines 1355-1360. Reproduce the network-connect behavior with the Docker commands in the issue, then trace how NetworkingConfig/EndpointsConfig and published repl ports are handled. Done means additional networks no longer remove published ports, or the agent fails fast with a clear error and logs the network driver.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
backend, distributed-systems, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.