hashicorp / hashicorp/nomad

idea: eliminate the use of "pause" containers for docker networking

Open
#15,086 19 comments 5 reactions 0 assignees View on GitHub
theme/driver/docker theme/networking type/enhancement
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

When a task group is using `"bridge"` networking mode, Nomad needs a network namespace shared by all tasks in the group. These tasks are not necessarily all using the same task driver (e.g. could be `exec` + `docker`, etc.)

Unfortunately once the Docker task driver gets involved this becomes tricky, because Docker explicitly does not support creating a container in a pre-defined network namespace (https://github.com/moby/moby/issues/7455). So unlike with the `exec` driver where we use normal Linux tooling to manage and join tasks to a shared network namespace, we are at the mercy of what Docker tooling enables.

The `docker network create` tool AFAICT has no support for creating a network namespace. Instead each container in Docker always gets created with its own network namespace, and then you can ["link"](https://docs.docker.com/network/links/) one container to another afterwords. This is where the `pause` container comes in. Because Docker ties the network namespace to a container (i.e. running process), Nomad needs a container that will not be stopped/replaced throughout the lifespan of the task group.

The problem of course, is that Bad Things can still happen to that pause container, and that is believed to be the source of most occurrences of https://github.com/hashicorp/nomad/issues/6385. While we can make improvements around cleaning up those orphaned resources, there's a lingering desire to just eliminate this whole class of problems outright.

I'm not sure what that means for users of the `docker` driver, unless we find a clever solution to start Docker containers in an existing network namespace, one that is created and managed by Nomad like we do for other task drivers. One idea would be to better invest in and promote the [podman](https://developer.hashicorp.com/nomad/plugins/drivers/podman) driver, which can also run Docker containers while being compatible with normal Linux tools and conventions.

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue names no files, tests, or entry points. Start by tracing Nomad's Docker-driver networking and pause-container lifecycle, then compare that behavior with the exec and Podman approaches described here; done would require a concrete, validated design that removes the pause-container dependency without breaking shared task-group networking.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker
Domain
infrastructure, networking
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
22/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.