aws / aws/aws-parallelcluster

(3.x.x) Instance Bootstrap Failures and Protected Mode when Launching Large Number of Nodes

Open
#7,265 0 comments 0 reactions 0 assignees View on GitHub
Backlog Bug
Dominant language
Python
Stars
888
Forks
314
Avg merge
1d 10h
Merged PRs (30d)
43

Description

## The issue
When launching a large number of instances, bootstrap failures may occur due to a delay in [EC2 API eventual consistency](https://docs.aws.amazon.com/ec2/latest/devguide/eventual-consistency.html).
When this happens, the private IP addresses might not be available in the response of `DescribeInstances` API, even though the ips have been assigned at the response of `RunInstances` API.

This can be seen in the clustermgtd logs:

```
2026-03-04 17:50:20,107 - [slurm_plugin.instance_manager:get_cluster_instances] - WARNING - Ignoring instance i-1234abcd because not all EC2 info are available, exception: KeyError, message: 'PrivateIpAddress'
```

The cluster treats this as a bootstrap failure and can enter protected mode and fail cluster creation. For information on protected mode and how to recover from it, refer to this [documentation](https://docs.aws.amazon.com/parallelcluster/latest/ug/slurm-protected-mode-v3.html).

## Affected ParallelCluster versions, OSes and schedulers
* All ParallelCluster versions and OSes
* Slurm scheduler

## Mitigation

You can find a detailed explanation and the mitigation of the problem [here](https://github.com/aws/aws-parallelcluster/wiki/(3.x.x)-Instance-Bootstrap-Failures-and-Protected-Mode-when-Launching-Large-Number-of-Nodes).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.