sysbox-fs zombie reaper can steal fusermount3 children during concurrent container creation
Nobody has claimed this yet.
- Dominant language
- Shell
- Stars
- 3.9k
- Forks
- 230
- Avg merge
- 7h 48m
- Merged PRs (30d)
- 3
Description
We have been seeing intermittent OCI container creation failures and my agent thinks the following is the cause:
Summary
Under concurrent container creation/deletion, Sysbox 0.7.1 intermittently fails container pre-registration because the sysbox-fs zombie reaper collects a fusermount3 child before the FUSE library's exec.Cmd.Wait() does.
Environment
- Sysbox: 0.7.1
- sysbox-fs commit:
c3d2ebc65102e32e74e383675f03b45556326888 - Architecture: arm64
- Kernel: 6.12
- Runtime: containerd 2.2.7
- FUSE helper: fusermount3
Observed failure
The following messages occurred consecutively in the same sysbox-fs process:
reaper: reaped pid 328923
fusermount: waitid: no child processes
Container pre-registration error: unable to initialize fuseServer for container 8f86022534b6: FuseServer InitWait error
Container creation subsequently succeeded on retry.
Root cause
The nsenter zombie reaper invokes process-wide wait4(-1):
The pinned FUSE library independently launches fusermount3 and calls cmd.Wait():
FUSE startup does not participate in the reaper's synchronization. The global reaper can therefore collect the fusermount3 child first, causing cmd.Wait() to return ECHILD.
The FUSE library has an attempted no child processes workaround, but it is applied to cmd.Start() rather than cmd.Wait(). It also checks "wait: no child processes", while the observed error is "waitid: no child processes".
Additional state leak
ContainerPreRegister inserts the container into idTable and netns tracking before starting FUSE:
When CreateFuseServer fails, that state and the created mountpoint directory are not rolled back:
The failed container's empty /var/lib/sysboxfs/<container-id> directory remained after the error.
Related concurrency issue
ContainerPreRegister holds the global container-state write lock throughout synchronous FUSE startup. During a burst of registrations, later requests can remain queued long enough for runtime/CNI cleanup to remove their network namespace:
Container pre-registration error: <id> has invalid net-ns:
Error getting netns inode: stat /var/run/netns/cni-...: no such file or directory
The gRPC cancellation context is also not propagated into the callback, so pre-registration may continue after its caller has abandoned container creation.
Suggested fixes
- Prevent the global zombie reaper from collecting children owned by
exec.Cmd, includingfusermount3. - Roll back
idTable, netns tracking, mountpoint, and partial FUSE state when pre-registration fails. - Avoid holding the global container DB lock during FUSE process startup.
- Pin or inspect the netns before waiting for the global lock.
- Propagate gRPC cancellation through pre-registration.
This is related to #981. PR nestybox/sysbox-fs#113 prevents the entire daemon from deadlocking when FUSE startup fails, but it does not prevent the reaper from stealing the FUSE helper or clean up failed pre-registration state.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read nsenter/reaper.go and state/containerDB.go first, focusing on the process-wide wait4 call, ContainerPreRegister, and CreateFuseServer failure paths. Compare the FUSE child handling in mount_linux.go. Done means concurrent creation no longer lets the reaper steal the FUSE child, and failed pre-registration leaves no container or mountpoint state behind.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- infrastructure, operating-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100