aws-eks: max HelmChart Timeout exceeds Lambda Duration
- Dominant language
- TypeScript
- Stars
- 12.9k
- Forks
- 4.6k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 83
Description
### Describe the bug
The HelmChart construct allows a user to set a Timeout which is passed to the Helm command along with the `--wait` option so that Helm will wait for the`timeout` to elapse before marking the operation as failed. This timeout can be set to a maximum of 15 minutes, which is the same timeout which is given to the kubectl custom resource provider Lambda Function.
However, there is initialization latency that results in the Lambda timeout (15m) expiring before the Helm operation reaches it's timeout (also 15m). This results in a severed cluster connection which in turn results in orphaned helm operations that become stuck in the `pending-upgrade` state. This is a tough state to resolve which typically requires a user to manually edit a Helm release secret or to rollback the release (see [this issue in the Helm Repo](https://github.com/helm/helm/issues/7476) for context).
### Expected Behavior
Expect timeouts to align in a cooperative manner such that the Lambda will not timeout before the HelmChart can return a response.
### Current Behavior
Lambda will timeout first when HelmChart is given max timeout resulting in a severed connection and tainted helm installation.
### Reproduction Steps
1. choose or create an invalid helm installation
2. create the cdk HelmChart construct with max `timeout` prop (`Duration.minutes(15)`)
3. cdk deploy
```ts
new eks.HelmChart(stack, 'MyFailingChart', { cluster, chart: 'chart', wait: true, timeout: Duration.minutes(15) });
```
### Possible Solution
Limit the max timeout for the HelmChart custom resource to be 870s (14.5m) instead of 900s to allow for initialization etc.
https://github.com/aws/aws-cdk/blob/main/packages/%40aws-cdk/aws-eks/lib/helm-chart.ts#L114-L116
### Additional Information/Context
_No response_
### CDK CLI Version
2.39.1 (build f188fac)
### Framework Version
2.28.1
### Node.js Version
v16.14.0
### OS
MacOS
### Language
Typescript
### Language Version
_No response_
### Other information
Related to #22254
Contributor guide
Research direction
Start at packages/@aws-cdk/aws-eks/lib/helm-chart.ts around lines 114-116, where the HelmChart timeout limit is defined. Check how the timeout is validated and covered by existing tests; done means the maximum allows Lambda initialization time before Helm expires, with the relevant test suite passing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, helm, kubernetes, typescript
- Domain
- cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100