Startup hang in oversubscribed containerized environments
- Dominant language
- C
- Stars
- 110
- Forks
- 35
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 2
Description
During Github Actions CI/CD testing, the first test occasionally hangs. Nick reproduced this in his own Docker environment. Bumping the number of containers up to 32 in Github more or less guarantees multiple ranks will hang. No OOM activity observed in dmesg logs. gdb backtraces isolated the problem to certain ranks not seeing the decorated `shell.init` message arriving in the event log.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the first-test hang in GitHub Actions or Nick's Docker environment, increasing the container count to 32. Inspect the gdb backtraces and event log around the decorated `shell.init` message, comparing ranks that receive it with those that do not. Done means startup no longer hangs under the oversubscribed container test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, docker, github-actions
- Domain
- devops, distributed-systems, hpc
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 38/100