aws / aws/aws-cdk

aws-eks: max HelmChart Timeout exceeds Lambda Duration

Open
#22,257 4 comments 2 reactions 0 assignees View on GitHub
@aws-cdk/aws-eks @aws-cdk/aws-eks-v2-alpha bug effort/small p2
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

### Describe the bug

The HelmChart construct allows a user to set a Timeout which is passed to the Helm command along with the `--wait` option so that Helm will wait for the`timeout` to elapse before marking the operation as failed. This timeout can be set to a maximum of 15 minutes, which is the same timeout which is given to the kubectl custom resource provider Lambda Function.
However, there is initialization latency that results in the Lambda timeout (15m) expiring before the Helm operation reaches it's timeout (also 15m). This results in a severed cluster connection which in turn results in orphaned helm operations that become stuck in the `pending-upgrade` state. This is a tough state to resolve which typically requires a user to manually edit a Helm release secret or to rollback the release (see [this issue in the Helm Repo](https://github.com/helm/helm/issues/7476) for context).

### Expected Behavior

Expect timeouts to align in a cooperative manner such that the Lambda will not timeout before the HelmChart can return a response.

### Current Behavior

Lambda will timeout first when HelmChart is given max timeout resulting in a severed connection and tainted helm installation.

### Reproduction Steps

1. choose or create an invalid helm installation
2. create the cdk HelmChart construct with max `timeout` prop (`Duration.minutes(15)`)
3. cdk deploy

```ts
new eks.HelmChart(stack, 'MyFailingChart', { cluster, chart: 'chart', wait: true, timeout: Duration.minutes(15) });
```

### Possible Solution

Limit the max timeout for the HelmChart custom resource to be 870s (14.5m) instead of 900s to allow for initialization etc.

https://github.com/aws/aws-cdk/blob/main/packages/%40aws-cdk/aws-eks/lib/helm-chart.ts#L114-L116

### Additional Information/Context

_No response_

### CDK CLI Version

2.39.1 (build f188fac)

### Framework Version

2.28.1

### Node.js Version

v16.14.0

### OS

MacOS

### Language

Typescript

### Language Version

_No response_

### Other information

Related to #22254

Contributor guide

Open the contributing guide

Research direction

Start at packages/@aws-cdk/aws-eks/lib/helm-chart.ts around lines 114-116, where the HelmChart timeout limit is defined. Check how the timeout is validated and covered by existing tests; done means the maximum allows Lambda initialization time before Helm expires, with the relevant test suite passing.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, helm, kubernetes, typescript
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.