docker / docker/cli

tcp_fin_timeout is not utilized from host kernel

Open
#4,100 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area/networking kind/bug status/0-triage
Dominant language
Go
Stars
6.1k
Forks
2.2k
Avg merge
1d 15h
Merged PRs (30d)
43

Description

Description

/proc/sys/net/ipv4/tcp_fin_timeout is not utilized when application connections within a docker container are sending a FIN_WAIT2.

The default should be a 60 seconds timeout. The file is completely missing inside a Docker container.

Also setting the timout by hand with sysctl does have no effect and returns a File oder directory not found

In our case where we have a small KVM-Qemu inside a Docker container with a TCP service.
Every client who disconnects will be CLOSED in KVM also on the host systen BUT NOT within Docker.

If we do a netstat -tunp we get a list of all disconnected clients still in FIN_WAIT2 state for example:

snip....
tcp        0      0 192.168.2.6:443         <client_ipv4>:47464      FIN_WAIT2   7/qemu-system-x86_6
tcp        0      0 192.168.2.6:443         <client_ipv4>:37036      FIN_WAIT2   7/qemu-system-x86_6
tcp        0      0 192.168.2.6:443         <client_ipv4>:50331      FIN_WAIT2   7/qemu-system-x86_6
tcp        0      0 192.168.2.6:443         <client_ipv4>:4387      FIN_WAIT2   7/qemu-system-x86_6
tcp        0      0 192.168.2.6:443         <client_ipv4>:27624      FIN_WAIT2   7/qemu-system-x86_6

These entries will never be cleared and at some point the Container crashes due to max open file descriptors.

Reproduce
  1. Create a Docker container with QEMU-KVM
  2. Create a service within the KVM guest (inside of Docker) such as a Webservice or any other service
  3. Forward ports from Host to Docker to KVM (for example: docker run ... -p 8080:80 ...)
  4. Use Userforward of KVM inside docker to redirect port 80 to KVM (for example: -net nic,model=virtio -net user,hostfwd=tcp::80-:80
  5. Connect to Docker by port 8080 and close the connection.
Expected behavior

Connections should be closed or terminated and cleared from Kernel FD table.

docker version
Client: Docker Engine - Community
 Version:           20.10.22
 API version:       1.41
 Go version:        go1.18.9
 Git commit:        3a2c30b
 Built:             Thu Dec 15 22:30:24 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.22
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.9
  Git commit:       42c8b31
  Built:            Thu Dec 15 22:28:33 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.15
  GitCommit:        5b842e528e99d4d4c1686467debf2bd4b88ecd86
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.9.1-docker)
  compose: Docker Compose (Docker Inc., v2.14.1)
  scan: Docker Scan (Docker Inc., v0.23.0)

Server:
 Containers: 13
  Running: 11
  Paused: 0
  Stopped: 2
 Images: 637
 Server Version: 20.10.22
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 5b842e528e99d4d4c1686467debf2bd4b88ecd86
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 3.10.0-1160.80.1.el7.x86_64
 Operating System: CentOS Linux 7 (Core)
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 251.6GiB
 Name: <hostname>
 ID: CRUH:W64O:O4N2:UY65:JSPP:DERI:4SVO:MLGK:ZHNA:MWGM:RHQ4:EBFD
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
Additional Info

Also with Docker v23.0.1 on CentOS 7, the result is the same.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the QEMU-in-Docker setup with the described docker run port forwarding and KVM user networking, then inspect /proc/sys/net/ipv4/tcp_fin_timeout and the FIN_WAIT2 entries reported by netstat. Compare the container and host behavior and determine whether the Docker CLI or another runtime component owns the missing setting; done means connections are cleared without exhausting file descriptors.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, linux
Domain
infrastructure, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.