llnl / llnl/Spindle

Startup hang in oversubscribed containerized environments

Open
#179 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
110
Forks
35
Avg merge
1d 16h
Merged PRs (30d)
2

Description

During Github Actions CI/CD testing, the first test occasionally hangs. Nick reproduced this in his own Docker environment. Bumping the number of containers up to 32 in Github more or less guarantees multiple ranks will hang. No OOM activity observed in dmesg logs. gdb backtraces isolated the problem to certain ranks not seeing the decorated `shell.init` message arriving in the event log.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the first-test hang in GitHub Actions or Nick's Docker environment, increasing the container count to 32. Inspect the gdb backtraces and event log around the decorated `shell.init` message, comparing ranks that receive it with those that do not. Done means startup no longer hangs under the oversubscribed container test.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, docker, github-actions
Domain
devops, distributed-systems, hpc
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.