opencontainers / opencontainers/runc
When the systemd is busy, runc init will hang and cannot exit, eventually ending up in D state.
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 13.5k
- Forks
- 2.3k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 30
Description
Description
In the scenario where a single node concurrently deploys more than 100 containers, the call chain is kubelet -> containerd -> containerd-shim-runc-v2 ->runc root create -> runc init -> dbus -> systemd -> cgroup. However, systemd is single-threaded and when it's busy, it continuously occupies one core. This causes the runc main process to be unable to get a response in a timely manner, with no timeout mechanism, leading to a deadlock. The related flame graph is as follows:
As shown, a main goroutine is blocked waiting for a message from a channel, while another goroutine, ReadMsgUnix, never receives a response, causing the main goroutine to block indefinitely.
The primary issue lies in the fact that runc init, as a client, doesn't incorporate a timeout mechanism. A potential solution would be to introduce a timeout, enabling runc to actively disconnect in the event of abnormal scenarios, such as systemd being overly busy. We could modify the context for the startUnit function in libcontainer/cgroups/systemd/common.go to include a timeout. Currently, the context for startUnit is context.TODO(), which has no timeout. We could change this to contextWithTimeout, _ := context.WithTimeout(context.Background(), 30*time.Second), ensuring that runc would not wait indefinitely in the face of unresponsive components.
Steps to reproduce the issue
- a single node deploys more than 100 pod at the same time with k8s
Describe the results you received and expected
- The process will not end up in a D (uninterruptible sleep) state.
What version of runc are you using?
1.1.2
Host OS information
x86
Host kernel information
Linux master1 5.10.0-60.18.0.50.h665.eulerosv2r11.x86_64 #1 SMP Fri Dec 23 16:12:27 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading libcontainer/cgroups/systemd/common.go, especially startUnit and its context handling, then trace how runc init waits for the systemd response. Reproduce the issue with the described concurrent container deployment if possible. Done means an unresponsive or overloaded systemd does not leave runc init waiting indefinitely or ending in D state.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, linux
- Domain
- cli, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100