(3.9.0-3.10.1) Cluster update intermittently fails because some compute nodes don’t execute update procedure
- Dominant language
- Python
- Stars
- 888
- Forks
- 314
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 43
Description
**The ParallelCluster team uses this template to report known issues on github. If you are reporting an issue, please use the 'Bug report' template instead.**
## Bug description
Cluster update intermittently fails:
```
$ pcluster list-clusters
{
"clusters": [
{
"clusterName": "test",
"cloudformationStackStatus": "UPDATE_ROLLBACK_COMPLETE",
"clusterStatus": "UPDATE_FAILED",
...
}
]
}
```
There is the following error in /var/log/chef-client.log on the head node:
```
================================================================================
Error executing action `run` on resource 'execute[Check cluster readiness]'
================================================================================
Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '1'
---- Begin output of /opt/parallelcluster/pyenv/versions/3.9.19/envs/cookbook_virtualenv/bin/python /opt/parallelcluster/scripts/head_node_checks/check_cluster_ready.py --cluster-name demo-cluster --table-name parallelcluster-demo-cluster --config-version 78dPxxb06z0XXX00hMwGxxxfzwxxPlYy --region us-west-2 ----
STDOUT:
STDERR: INFO:__main__:Checking cluster readiness with arguments: cluster_name=demo-cluster, table_name=parallelcluster-demo-cluster, config_version=78dPxxb06z0XXX00hMwGxxxfzwxxPlYy, region=us-west-2
INFO:__main__:Checking that cluster configuration deployed on cluster nodes for cluster demo-cluster is 78dPxxb06z0XXX00hMwGxxxfzwxxPlYy
INFO:botocore.credentials:Found credentials from IAM Role: demo-cluster-RoleHeadNode-xxxx
INFO:__main__:Found batch of 4 cluster node(s): ['i-xxxxxxxxxxxxxxxxx', 'i-yyyyyyyyyyyyyyyyy', 'i-aaaaaaaaaaaaaaaaa', 'i-bbbbbbbbbbbbbbbbb']
INFO:__main__:Retrieved 4 DDB item(s):
{'Id': {'S': 'CLUSTER_CONFIG.i-xxxxxxxxxxxxxxxxx'}, 'Data': {'M': {'node_type': {'S': 'ComputeFleet'}, 'cluster_config_version': {'S': '78dPxxb06z0XXX00hMwGxxxfzwxxPlYy'}, 'status': {'S': 'DEPLOYED'}, 'lastUpdateTime': {'S': '2024-08-02 22:27:50 UTC'}}}}
{'Id': {'S': 'CLUSTER_CONFIG.i-yyyyyyyyyyyyyyyyy'}, 'Data': {'M': {'node_type': {'S': 'ComputeFleet'}, 'cluster_config_version': {'S': '1_6PjwxxxvWZNZtxxBGxxxRQkVdTGqft'}, 'status': {'S': 'DEPLOYED'}, 'lastUpdateTime': {'S': '2024-08-01 16:58:37 UTC'}}}}
{'Id': {'S': 'CLUSTER_CONFIG.i-aaaaaaaaaaaaaaaaa'}, 'Data': {'M': {'node_type': {'S': 'ComputeFleet'}, 'cluster_config_version': {'S': '78dPxxb06z0XXX00hMwGxxxfzwxxPlYy'}, 'status': {'S': 'DEPLOYED'}, 'lastUpdateTime': {'S': '2024-08-02 22:27:38 UTC'}}}}
{'Id': {'S': 'CLUSTER_CONFIG.i-bbbbbbbbbbbbbbbbb'}, 'Data': {'M': {'node_type': {'S': 'ComputeFleet'}, 'cluster_config_version': {'S': '1_6PjwxxxvWZNZtxxBGxxxRQkVdTGqft'}, 'status': {'S': 'DEPLOYED'}, 'lastUpdateTime': {'S': '2024-08-01 16:58:33 UTC'}}}}
ERROR:__main__:Some cluster readiness checks failed: Check failed due to the following erroneous records:
* missing records (0): []
* incomplete records (0): []
* wrong records (2): [('i-yyyyyyyyyyyyyyyyy', '1_6PjwxxxvWZNZtxxBGxxxRQkVdTGqft'), ('i-bbbbbbbbbbbbbbbbb', '1_6PjwxxxvWZNZtxxBGxxxRQkVdTGqft')]
```
## Affected versions (OSes, schedulers)
* ParallelCluster 3.9.0-3.10.1
* Slurm scheduler
* All operating systems
## Mitigation
See details in [Wiki](https://github.com/aws/aws-parallelcluster/wiki/(3.9.0%E2%80%903.10.1)-Cluster-update-intermittently-fails-because-some-compute-nodes-don%E2%80%99t-execute-update-procedure)
Contributor guide
Assessment
This issue has not been assessed yet.