nestybox / nestybox/sysbox

sysbox-fs zombie reaper can steal fusermount3 children during concurrent container creation

Open
#1,044 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Shell
Stars
3.9k
Forks
230
Avg merge
7h 48m
Merged PRs (30d)
3

Description

We have been seeing intermittent OCI container creation failures and my agent thinks the following is the cause:

Summary

Under concurrent container creation/deletion, Sysbox 0.7.1 intermittently fails container pre-registration because the sysbox-fs zombie reaper collects a fusermount3 child before the FUSE library's exec.Cmd.Wait() does.

Environment

  • Sysbox: 0.7.1
  • sysbox-fs commit: c3d2ebc65102e32e74e383675f03b45556326888
  • Architecture: arm64
  • Kernel: 6.12
  • Runtime: containerd 2.2.7
  • FUSE helper: fusermount3

Observed failure

The following messages occurred consecutively in the same sysbox-fs process:

reaper: reaped pid 328923
fusermount: waitid: no child processes
Container pre-registration error: unable to initialize fuseServer for container 8f86022534b6: FuseServer InitWait error

Container creation subsequently succeeded on retry.

Root cause

The nsenter zombie reaper invokes process-wide wait4(-1):

https://github.com/nestybox/sysbox-fs/blob/c3d2ebc65102e32e74e383675f03b45556326888/nsenter/reaper.go#L88-L97

The pinned FUSE library independently launches fusermount3 and calls cmd.Wait():

https://github.com/nestybox/fuse/blob/5ce319439091b08f33b149cb2482039f1b0a24fe/mount_linux.go#L74-L107

FUSE startup does not participate in the reaper's synchronization. The global reaper can therefore collect the fusermount3 child first, causing cmd.Wait() to return ECHILD.

The FUSE library has an attempted no child processes workaround, but it is applied to cmd.Start() rather than cmd.Wait(). It also checks "wait: no child processes", while the observed error is "waitid: no child processes".

Additional state leak

ContainerPreRegister inserts the container into idTable and netns tracking before starting FUSE:

https://github.com/nestybox/sysbox-fs/blob/c3d2ebc65102e32e74e383675f03b45556326888/state/containerDB.go#L132-L176

When CreateFuseServer fails, that state and the created mountpoint directory are not rolled back:

https://github.com/nestybox/sysbox-fs/blob/c3d2ebc65102e32e74e383675f03b45556326888/state/containerDB.go#L176-L186

The failed container's empty /var/lib/sysboxfs/<container-id> directory remained after the error.

Related concurrency issue

ContainerPreRegister holds the global container-state write lock throughout synchronous FUSE startup. During a burst of registrations, later requests can remain queued long enough for runtime/CNI cleanup to remove their network namespace:

Container pre-registration error: <id> has invalid net-ns:
Error getting netns inode: stat /var/run/netns/cni-...: no such file or directory

The gRPC cancellation context is also not propagated into the callback, so pre-registration may continue after its caller has abandoned container creation.

Suggested fixes

  1. Prevent the global zombie reaper from collecting children owned by exec.Cmd, including fusermount3.
  2. Roll back idTable, netns tracking, mountpoint, and partial FUSE state when pre-registration fails.
  3. Avoid holding the global container DB lock during FUSE process startup.
  4. Pin or inspect the netns before waiting for the global lock.
  5. Propagate gRPC cancellation through pre-registration.

This is related to #981. PR nestybox/sysbox-fs#113 prevents the entire daemon from deadlocking when FUSE startup fails, but it does not prevent the reaper from stealing the FUSE helper or clean up failed pre-registration state.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read nsenter/reaper.go and state/containerDB.go first, focusing on the process-wide wait4 call, ContainerPreRegister, and CreateFuseServer failure paths. Compare the FUSE child handling in mount_linux.go. Done means concurrent creation no longer lets the reaper steal the FUSE child, and failed pre-registration leaves no container or mountpoint state behind.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
infrastructure, operating-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.