aws / aws/aws-parallelcluster

Cluster update times out while fetching compute node LaunchTemplates for queues

Open
#7,519 2 comments 1 reaction 0 assignees View on GitHub
Backlog
Dominant language
Python
Stars
888
Forks
314
Avg merge
1d 10h
Merged PRs (30d)
43

Description

## Required Info

- AWS ParallelCluster version: 3.15.0
- OS: Ubuntu 24.04 LTS
- Region: us-east-1

## Bug description and how to reproduce

When queue settings are changed and the cluster is updated with `pcluster update-cluster`, the process times out while the Chef run on the head node retrieves the Compute Node LaunchTemplates associated with the queues.

We believe this occurs because the current implementation assumes that processing all queues will complete within 30 seconds.

As a temporary workaround, we changed the timeout setting from `30` to `300` directly in the following file:

- `/etc/chef/cookbooks/aws-parallelcluster-platform/resources/fetch_dna_files.rb`

## Expected behavior

As the number of queues increases, simply extending the timeout does not provide a fundamental solution. The timeout should apply to the processing of each queue rather than to the processing of all queues together.

## Related Issues

- [Issue #7203: HeadNode checks all compute nodes when adding queue, causing timeout with running jobs](https://github.com/aws/aws-parallelcluster/issues/7203)

Contributor guide

Open the contributing guide

Research direction

Start with /etc/chef/cookbooks/aws-parallelcluster-platform/resources/fetch_dna_files.rb and trace how pcluster update-cluster retrieves Compute Node LaunchTemplates for queues. Reproduce the update with multiple queues if possible, then verify that the timeout is applied per queue rather than to processing all queues together.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, ruby
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.