moby / moby/libnetwork

iptables locking issues if --wait is not available

Open
#2,625 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
2.2k
Forks
875
PR merge metrics
No merged PRs in 30d

Description

If an iptables version is used that does not support or expose (in case a wrapper script is used arround iptables) the --wait flag issues occur if multiple containers are started at the same time. As only one iptables instance can run at any time, parallel calls to iptables fail and the rules for the dns resolver are not added to the container (in case a docker network is used).

Symptom:
Containers can not access other containers or the internet due to the DNS resolution failing.

Steps to reproduce:

  • Use an iptables wrapper or version without --wait
  • Create a docker network
  • Add multiple containers to this network that are restarted on system start
  • Restart docker or the system to start all containers at once

Cause for this issue:
Docker will detect that the xtables lock (iptables/iptables.go:121) is not supported and use its internal locking mechanism (bestEffortLock) instead (iptables/iptables.go:537).

The issue is that this lock is a mutex and can only provide locking within the same process. This becomes an issue if docker networks are used. In this case a new resolver process for each container is spawned via "reexec" to add the required iptables rules into the container network namespace (resolver_unix.go:29). If multiple containers start in parallel, multiple of these resolver setup processes are spawned which each have their own independent bestEffortLock and thus try to execute iptables in parallel to each other - and fail while attempting to do this.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with iptables/iptables.go at the xtables-lock detection and bestEffortLock references, then inspect resolver_unix.go where reexec processes add iptables rules. Reproduce concurrent container startup with an iptables wrapper or version lacking --wait. Done means parallel resolver setup no longer causes iptables calls to fail and the DNS rules are added to each container network.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.