(3.6.0 - latest) Prolog hangs due to long GPU health check times on certain instance types
- Dominant language
- Python
- Stars
- 888
- Forks
- 314
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 43
Description
## The Issue
AWS ParallelCluster 3.6.0 and later, when configured with GPU health checks, may experience delays and eventual "prolog hung" errors when using instance types like p4d and p5. These instances have a complex GPU topology, which results in lengthy diagnostic checks during the GPU health check process. The Prolog, which runs the check before job tasks are started, must complete on all allocated nodes. If one node's GPU health check takes too long, the entire job setup is delayed, causing the "prolog hung" error:
```
slurmstepd: error: Prolog hung on node xxx
```
Testing with g4dn and g6 instance types shows that this issue occurs rarely, as these instances have simpler GPU configurations that do not require such long diagnostic times.
## Affected Versions
All ParallelCluster 3.6.0+ versions using Slurm scheduler on instance types such as p4d and p5 with [GPU health checks enabled](https://docs.aws.amazon.com/parallelcluster/latest/ug/Scheduling-v3.html#yaml-Scheduling-SlurmQueues-ComputeResources-HealthChecks-Gpu-Enabled) are affected. This issue may also occur with other instance types if the GPU health check takes too long.
## Mitigation
You can find a detailed explanation and the mitigation of the problem. [(3.6.0 ‐ latest) Long GPU health checks Prolog cause “Prolog hung” or “Batch Job missing” errors](https://github.com/aws/aws-parallelcluster/wiki/(3.6.0-%E2%80%90-latest)-Long-GPU-health-checks---Prolog-cause-%E2%80%9CProlog-hung%E2%80%9D-or-%E2%80%9CBatch-Job-missing%E2%80%9D-errors)
Contributor guide
Research direction
Start with the linked wiki mitigation and AWS GPU health-check documentation; compare behavior on p4d/p5 versus g4dn/g6 with Slurm GPU health checks enabled. The issue names no source file, test, or requested code change, so completion criteria must be clarified before implementation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100