Cluster update times out while fetching compute node LaunchTemplates for queues
- Dominant language
- Python
- Stars
- 888
- Forks
- 314
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 43
Description
## Required Info
- AWS ParallelCluster version: 3.15.0
- OS: Ubuntu 24.04 LTS
- Region: us-east-1
## Bug description and how to reproduce
When queue settings are changed and the cluster is updated with `pcluster update-cluster`, the process times out while the Chef run on the head node retrieves the Compute Node LaunchTemplates associated with the queues.
We believe this occurs because the current implementation assumes that processing all queues will complete within 30 seconds.
As a temporary workaround, we changed the timeout setting from `30` to `300` directly in the following file:
- `/etc/chef/cookbooks/aws-parallelcluster-platform/resources/fetch_dna_files.rb`
## Expected behavior
As the number of queues increases, simply extending the timeout does not provide a fundamental solution. The timeout should apply to the processing of each queue rather than to the processing of all queues together.
## Related Issues
- [Issue #7203: HeadNode checks all compute nodes when adding queue, causing timeout with running jobs](https://github.com/aws/aws-parallelcluster/issues/7203)
Contributor guide
Research direction
Start with /etc/chef/cookbooks/aws-parallelcluster-platform/resources/fetch_dna_files.rb and trace how pcluster update-cluster retrieves Compute Node LaunchTemplates for queues. Reproduce the update with multiple queues if possible, then verify that the timeout is applied per queue rather than to processing all queues together.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, ruby
- Domain
- cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100