aws / aws/aws-cdk

(@aws-cdk/aws-eks-v2-alpha): kubectl handler masks error messages with "Response object is too long" when kubectl output exceeds 4KB

Open Beginner friendly
#37,353 2 comments 0 reactions 0 assignees View on GitHub
@aws-cdk/aws-eks-v2 bug effort/small p2
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
1d 19h
Merged PRs (30d)
74

Description

### Describe the bug

The CDK kubectl handler Lambda (lib/kubectl-handler/apply/__init__.py) passes raw kubectl error output as the exception message without truncation. When kubectl apply fails with a validation error, Kubernetes includes the entire object being validated in its error output. This often exceeds CloudFormation's 4KB custom resource response limit, causing CloudFormation to replace the actual error with the unhelpful message: Response object is too long.

The real error is only visible in the Lambda's CloudWatch logs.

### Regression Issue

- [ ] Select this option if this issue appears to be a regression.

### Last Known Working CDK Library Version

_No response_

### Expected Behavior

When kubectl apply fails, the actual error message (e.g., strict decoding error: unknown field "spec.terminationGracePeriod") should be visible in the CloudFormation stack events, not hidden behind Response object is too long.

### Current Behavior

CloudFormation shows only `Response object is too long.` because the kubectl handler raises the full kubectl output (which includes the entire K8s object JSON) as the exception. The CloudFormation custom resource framework serializes this into the response Reason field, exceeding the 4KB response payload limit.

Error in CloudWatch logs (truncated for readability):
```
Exception: b'The request is invalid: patch: Invalid value: "{...entire object JSON...}": strict decoding error: unknown field "spec.terminationGracePeriod"'
```

### Reproduction Steps

```typescript
import * as eks from '@aws-cdk/aws-eks-v2-alpha';

// Given an existing EKS cluster
const cluster = new eks.Cluster(this, 'Cluster', { ... });

// Apply a manifest with an intentionally invalid field
cluster.addManifest('test', {
apiVersion: 'karpenter.sh/v1',
kind: 'NodePool',
metadata: { name: 'test' },
spec: {
// This field doesn't exist at spec level (should be spec.template.spec)
terminationGracePeriod: '48h',
template: {
spec: {
nodeClassRef: { group: 'eks.amazonaws.com', kind: 'NodeClass', name: 'default' },
requirements: [{ key: 'karpenter.sh/capacity-type', operator: 'In', values: ['on-demand'] }],
},
},
limits: { cpu: '100' },
disruption: { consolidationPolicy: 'WhenEmpty', consolidateAfter: '1h' },
},
});
```

Deploy, CloudFormation reports Response object is too long instead of the schema validation error.

### Possible Solution

Truncate the error message in lib/kubectl-handler/apply/__init__.py line 89:
```python
# Current
raise Exception(output)

# Fix
MAX_ERROR_LENGTH = 2048 # Leave room for other response fields within 4KB limit
error_msg = output.decode('utf-8', errors='replace')[:MAX_ERROR_LENGTH]
raise Exception(error_msg)
```
The same issue exists in line 93 (`raise Exception(f'Operation failed after {maxAttempts} attempts: {output}')`).

### Additional Information/Context

The `patch/__init__.py` handler has a similar pattern and would benefit from the same fix.

### AWS CDK Library version (aws-cdk-lib)

@aws-cdk/aws-eks-v2-alpha

### AWS CDK CLI version

2.x

### Node.js Version

18.x

### OS

Amazon Linux 2

### Language

TypeScript

### Language Version

_No response_

### Other information

The 4KB limit is a hard constraint on CloudFormation custom resource responses. Kubernetes validation errors are particularly verbose because they include the full object being validated in the error message, making this a common issue for any non-trivial manifest.

Contributor guide

Open the contributing guide

Research direction

Start in lib/kubectl-handler/apply/__init__.py around lines 89 and 93, then compare the similar error handling in patch/__init__.py. Trace both failure paths and verify that verbose kubectl errors are bounded while retaining the useful validation message and staying within CloudFormation's 4KB custom-resource response limit.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes, python
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.