opencontainers / opencontainers/runc

runc hang on init when containerd set up

Open
#4,481 17 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
13.5k
Forks
2.3k
Avg merge
2d 8h
Merged PRs (30d)
30

Description

Description

I find some D state process on node which containerd set up

5 D root      14378  13862  0  80   0 - 269979 refrig Sep30 ?       00:00:00 /usr/local/bin/runc init
5 D root      14392  13587  0  80   0 - 270107 refrig Sep30 ?       00:00:00 /usr/local/bin/runc init
0 S root     278169 276735  0  80   0 -  1007 pipe_r 00:44 pts/2    00:00:00 grep --color=auto  D 
root@hsotname:~# cat /proc/14378/stack 
[<0>] __refrigerator+0x4c/0x130
[<0>] unix_stream_data_wait+0x1fa/0x210
[<0>] unix_stream_read_generic+0x50d/0xa60
[<0>] unix_stream_recvmsg+0x88/0x90
[<0>] sock_recvmsg+0x70/0x80
[<0>] sock_read_iter+0x8f/0xf0
[<0>] new_sync_read+0x180/0x190
[<0>] vfs_read+0xff/0x1a0
[<0>] ksys_read+0xb1/0xe0
[<0>] __x64_sys_read+0x19/0x20
[<0>] do_syscall_64+0x5c/0xc0
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xae
root@hostname:~# uptime
 01:32:12 up 28 days, 38 min,  2 users,  load average: 29.57, 31.53, 31.98
root@hostname:~# systemctl status containerd
● containerd.service - containerd container runtime
     Loaded: loaded (/etc/systemd/system/containerd.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2024-09-30 00:53:45 -07; 4 weeks 0 days ago
root@hostname~# ps -eo pid,lstart,cmd,state |grep 14378
 14378 Mon Sep 30 00:53:38 2024 /usr/local/bin/runc init    D
root@hostname:~# stat /var/containerd/containerd.sock
  File: /var/containerd/containerd.sock
  Size: 0               Blocks: 0          IO Block: 4096   socket
Device: 10303h/66307d   Inode: 1082291752  Links: 1
Access: (0660/srw-rw----)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-10-28 00:45:08.361633324 -0700
Modify: 2024-09-30 00:53:45.666038162 -0700
Change: 2024-09-30 00:53:45.666038162 -0700
 Birth: 2024-09-30 00:53:45.666038162 -0700

The runc init process set up before /var/containerd/containerd.sock changed. I think there is something race on it? But i think the runc process should wait timeout and exit.

Steps to reproduce the issue

No response

Describe the results you received and expected

The runc init hang. Expected no D state process.

What version of runc are you using?

~# runc --version
runc version 1.1.2
commit: c4f88bc9
spec: 1.0.2-dev
go: go1.17.13
libseccomp: 2.5.3

Host OS information

~# cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.5 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
BUILD_ID="ubuntu-240918-061134"

Host kernel information

~# uname -a
Linux tess-node-ttbts-tess134.stratus.lvs.ebay.com 5.15.0-26-generic #26 SMP Wed Sep 18 09:16:49 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the /usr/local/bin/runc init entry point, the /var/containerd/containerd.sock state, and the captured kernel stack showing unix_stream_read_generic. Review the 17-comment discussion and investigate the reported runc 1.1.2 behavior on Ubuntu 22.04 with kernel 5.15. Done means identifying the hang or race and defining behavior that avoids a persistent D-state process.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, linux, ubuntu
Domain
devops, infrastructure, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.