(@aws-cdk/aws-eks-v2-alpha): kubectl handler masks error messages with "Response object is too long" when kubectl output exceeds 4KB
- Dominant language
- TypeScript
- Stars
- 12.9k
- Forks
- 4.6k
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 74
Description
### Describe the bug
The CDK kubectl handler Lambda (lib/kubectl-handler/apply/__init__.py) passes raw kubectl error output as the exception message without truncation. When kubectl apply fails with a validation error, Kubernetes includes the entire object being validated in its error output. This often exceeds CloudFormation's 4KB custom resource response limit, causing CloudFormation to replace the actual error with the unhelpful message: Response object is too long.
The real error is only visible in the Lambda's CloudWatch logs.
### Regression Issue
- [ ] Select this option if this issue appears to be a regression.
### Last Known Working CDK Library Version
_No response_
### Expected Behavior
When kubectl apply fails, the actual error message (e.g., strict decoding error: unknown field "spec.terminationGracePeriod") should be visible in the CloudFormation stack events, not hidden behind Response object is too long.
### Current Behavior
CloudFormation shows only `Response object is too long.` because the kubectl handler raises the full kubectl output (which includes the entire K8s object JSON) as the exception. The CloudFormation custom resource framework serializes this into the response Reason field, exceeding the 4KB response payload limit.
Error in CloudWatch logs (truncated for readability):
```
Exception: b'The request is invalid: patch: Invalid value: "{...entire object JSON...}": strict decoding error: unknown field "spec.terminationGracePeriod"'
```
### Reproduction Steps
```typescript
import * as eks from '@aws-cdk/aws-eks-v2-alpha';
// Given an existing EKS cluster
const cluster = new eks.Cluster(this, 'Cluster', { ... });
// Apply a manifest with an intentionally invalid field
cluster.addManifest('test', {
apiVersion: 'karpenter.sh/v1',
kind: 'NodePool',
metadata: { name: 'test' },
spec: {
// This field doesn't exist at spec level (should be spec.template.spec)
terminationGracePeriod: '48h',
template: {
spec: {
nodeClassRef: { group: 'eks.amazonaws.com', kind: 'NodeClass', name: 'default' },
requirements: [{ key: 'karpenter.sh/capacity-type', operator: 'In', values: ['on-demand'] }],
},
},
limits: { cpu: '100' },
disruption: { consolidationPolicy: 'WhenEmpty', consolidateAfter: '1h' },
},
});
```
Deploy, CloudFormation reports Response object is too long instead of the schema validation error.
### Possible Solution
Truncate the error message in lib/kubectl-handler/apply/__init__.py line 89:
```python
# Current
raise Exception(output)
# Fix
MAX_ERROR_LENGTH = 2048 # Leave room for other response fields within 4KB limit
error_msg = output.decode('utf-8', errors='replace')[:MAX_ERROR_LENGTH]
raise Exception(error_msg)
```
The same issue exists in line 93 (`raise Exception(f'Operation failed after {maxAttempts} attempts: {output}')`).
### Additional Information/Context
The `patch/__init__.py` handler has a similar pattern and would benefit from the same fix.
### AWS CDK Library version (aws-cdk-lib)
@aws-cdk/aws-eks-v2-alpha
### AWS CDK CLI version
2.x
### Node.js Version
18.x
### OS
Amazon Linux 2
### Language
TypeScript
### Language Version
_No response_
### Other information
The 4KB limit is a hard constraint on CloudFormation custom resource responses. Kubernetes validation errors are particularly verbose because they include the full object being validated in the error message, making this a common issue for any non-trivial manifest.
Contributor guide
Research direction
Start in lib/kubectl-handler/apply/__init__.py around lines 89 and 93, then compare the similar error handling in patch/__init__.py. Trace both failure paths and verify that verbose kubectl errors are bounded while retaining the useful validation message and staying within CloudFormation's 4KB custom-resource response limit.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes, python
- Domain
- cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100