custom-resource: versioned Custom Resource Handlers

WIP

# Feature Request: Versioned Custom Resource Handlers

## Describe the feature

Implement content-based versioning for CDK custom resource handlers so that existing deployments continue using their original handler version while new deployments automatically receive the latest version. This would prevent unintended handler updates during stack updates and enable safer rollout of handler improvements.

## Use Case

**I'm always frustrated when** I upgrade my CDK version and my production stacks automatically receive updated custom resource handler code without my knowledge or consent, potentially introducing breaking changes or regressions.

### The Core Problem

When CDK custom resource handlers are modified and released in a new CDK version, all existing deployments automatically receive the updated handler code during their next stack update. This creates several critical issues:

1. **Unexpected Breaking Changes**: Handler modifications can introduce breaking changes to existing stacks without user awareness
2. **No Opt-Out Mechanism**: Users cannot pin to a specific handler version that works for their deployment
3. **Forced Upgrades**: Stack updates unrelated to the custom resource still trigger handler updates
4. **Risk of Regression**: Bug fixes or refactors in handlers can inadvertently break existing deployments
5. **Innovation Blocked**: Fear of breaking existing deployments prevents rolling out important improvements and features

### Real-World Impact: Blocked Features

This limitation has prevented or significantly delayed important features from being rolled out:

#### 1. S3 Deployment Custom Resource Improvements ([#7090](https://github.com/aws/aws-cdk/issues/7090))
The S3 deployment custom resource handler could benefit from numerous improvements:
- Better error handling and retry logic
- Support for additional S3 features (storage classes, metadata, etc.)
- Performance optimizations for large deployments
- Enhanced logging and debugging capabilities

However, any changes risk breaking existing deployments that rely on current behavior, making the CDK team hesitant to make improvements.

#### 2. EKS kubectl Custom Resources
The kubectl handler used for EKS cluster management faces similar constraints:
- Cannot easily upgrade kubectl versions (might break existing manifests)
- Difficult to add new Kubernetes API support
- Improvements to RBAC handling blocked by compatibility concerns
- Enhanced error messages and debugging features delayed

#### 3. Other Affected Custom Resources
- **Lambda Layer Version Handler**: Cannot optimize deployment strategies
- **ECR Image Asset Handler**: Improvements to image scanning and tagging blocked
- **Route53 Cross-Account Handler**: Better error handling and retry logic delayed
- **CloudFront Invalidation Handler**: Performance improvements postponed

### The Innovation Dilemma

```
┌─────────────────────────────────────────────────────────────┐
│ Current State: Innovation vs. Stability Trade-off │
└─────────────────────────────────────────────────────────────┘

Feature Request or Bug Fix
│
▼
┌─────────────────────┐
│ Will this change │
│ handler behavior? │
└──────┬──────────────┘
│
┌─────┴─────┐
│ │
YES NO
│ │
▼ ▼
┌─────────┐ ┌──────────┐
│ BLOCKED │ │ APPROVED │
│ or │ │ │
│ DELAYED │ └──────────┘
└─────────┘
│
└─► Reason: Risk of breaking existing deployments
Result: Users miss out on improvements
```

With versioned handlers, this dilemma disappears:
- Existing deployments remain stable with their current handler version
- New deployments automatically benefit from improvements
- Users can opt-in to handler upgrades when ready

### Current Behavior (Problematic)

```
┌─────────────────────────────────────────────────────────────────┐
│ Timeline: CDK Version Upgrades │
└─────────────────────────────────────────────────────────────────┘

CDK v2.100.0 Released
│
├─► Handler Code: index.js (version A)
│
▼
┌──────────────────┐
│ User deploys │
│ Production │
│ Stack │
└────────┬─────────┘
│
▼
┌─────────────────────────────────┐
│ AWS Account (Production) │
│ │
│ ┌───────────────────────────┐ │
│ │ Lambda Function │ │
│ │ "CustomResourceHandler" │ │
│ │ Code: Handler Version A │ │
│ │ ✅ Working in production │ │
│ └───────────────────────────┘ │
└─────────────────────────────────┘

⏰ Time passes...

CDK v2.101.0 Released
│
├─► Handler Code: index.js (version B - MODIFIED)
│ • Bug fix applied
│ • Logic changed
│
▼
┌──────────────────┐
│ User runs │
│ cdk deploy │
│ (unrelated │
│ change to │
│ their stack) │
└────────┬─────────┘
│
▼
┌─────────────────────────────────┐
│ AWS Account (Production) │
│ │
│ ┌───────────────────────────┐ │
│ │ Lambda Function │ │
│ │ "CustomResourceHandler" │ │
│ │ Code: Handler Version B │◄─┼─── ⚠️ AUTOMATICALLY UPDATED!
│ │ ❌ Potential breaking │ │
│ │ change in production! │ │
│ └───────────────────────────┘ │
└─────────────────────────────────┘

Problem: User had no control over handler update!
```

## Proposed Solution

Implement **content-based versioning** for custom resource handlers, where each unique version of handler code is deployed as a separate Lambda function, and existing stacks continue using their original handler version.

### Desired Behavior (With Versioning)

```
┌─────────────────────────────────────────────────────────────────┐
│ Timeline: CDK Version Upgrades with Versioned Handlers │
└─────────────────────────────────────────────────────────────────┘

CDK v2.100.0 Released
│
├─► Handler Code: index.js
│ Hash: a1b2c3d4
│
▼
┌──────────────────┐
│ User deploys │
│ Production │
│ Stack │
└────────┬─────────┘
│
▼
┌─────────────────────────────────────────┐
│ AWS Account (Production) │
│ │
│ ┌───────────────────────────────────┐ │
│ │ Lambda Function │ │
│ │ "CustomResourceHandler-a1b2c3d4" │ │
│ │ Code: Handler Version A │ │
│ │ ✅ Working in production │ │
│ └───────────────────────────────────┘ │
└─────────────────────────────────────────┘

⏰ Time passes...

CDK v2.101.0 Released
│
├─► Handler Code: index.js (MODIFIED)
│ Hash: e5f6g7h8 (different!)
│
▼
┌──────────────────┐
│ User runs │
│ cdk deploy │
│ (unrelated │
│ change to │
│ their stack) │
└────────┬─────────┘
│
▼
┌─────────────────────────────────────────┐
│ AWS Account (Production) │
│ │
│ ┌───────────────────────────────────┐ │
│ │ Lambda Function │ │
│ │ "CustomResourceHandler-a1b2c3d4" │ │
│ │ Code: Handler Version A │ │
│ │ ✅ UNCHANGED - Still working! │◄─┼─── ✅ Existing stack safe!
│ └───────────────────────────────────┘ │
│ │
│ ┌───────────────────────────────────┐ │
│ │ Lambda Function │ │
│ │ "CustomResourceHandler-e5f6g7h8" │ │
│ │ Code: Handler Version B │ │
│ │ ⏸️ Available for new stacks │◄─┼─── ✅ New version isolated!
│ └───────────────────────────────────┘ │
└─────────────────────────────────────────┘

⏰ Later: User creates NEW stack

┌──────────────────┐
│ User deploys │
│ New Stack │
└────────┬─────────┘
│
▼
┌─────────────────────────────────────────┐
│ AWS Account (Production) │
│ │
│ ┌───────────────────────────────────┐ │
│ │ Lambda Function │ │
│ │ "CustomResourceHandler-a1b2c3d4" │ │
│ │ ✅ Old stack still uses this │ │
│ └───────────────────────────────────┘ │
│ │
│ ┌───────────────────────────────────┐ │
│ │ Lambda Function │ │
│ │ "CustomResourceHandler-e5f6g7h8" │ │
│ │ ✅ New stack uses this │◄─┼─── ✅ Automatic upgrade!
│ └───────────────────────────────────┘ │
└─────────────────────────────────────────┘

Benefits:
• Existing stacks: Stable and unchanged ✅
• New stacks: Automatically get latest version ✅
• User control: Explicit upgrade path available ✅
```

## Proposed Implementation

### 1. Content-Based Hashing

Generate a hash of the handler code and include it in the Lambda function's logical ID:

```typescript
// Before (current)
const handler = new lambda.SingletonFunction(this, 'CustomResourceHandler', {
uuid: 'custom-resource-handler',
code: lambda.Code.fromAsset(handlerPath),
// ...
});

// After (proposed)
const handlerHash = hashCode(fs.readFileSync(handlerPath)); // e.g., "a1b2c3d4"
const handler = new lambda.SingletonFunction(this, `CustomResourceHandler-${handlerHash}`, {
uuid: `custom-resource-handler-${handlerHash}`,
code: lambda.Code.fromAsset(handlerPath),
// ...
});
```

### 2. Feature Flag for Gradual Rollout

Introduce a feature flag to enable versioned handlers:

```json
{
"@aws-cdk/core:versionedCustomResourceHandlers": true
}
```

### 3. Backward Compatibility

- **Default behavior**: Disabled (maintains current behavior)
- **Opt-in**: Users can enable via feature flag
- **Future**: Enable by default in next major version

## Benefits

1. **Stability**: Existing deployments remain stable and unaffected by handler updates
2. **Safety**: Handler changes only affect new deployments or explicit upgrades
3. **Predictability**: Users know exactly when handler versions change
4. **Rollback Safety**: Old handler versions remain available if rollback is needed
5. **Testing**: New handler versions can be tested in isolation before wide adoption
6. **Innovation Unlocked**: CDK team can confidently improve handlers without fear of breaking existing deployments

## Other Information

### Alternative Approaches Considered

### Resource Management

- Multiple Lambda functions (one per handler version) will exist in the account
- CloudFormation will automatically clean up old handlers when stacks are deleted
- Singleton pattern ensures only one function per version per stack

### S3 Asset Management

- Handler code assets will be versioned in S3
- CDK bootstrap bucket will contain multiple versions
- Existing CDK asset cleanup mechanisms apply

### Migration Path

**Phase 1**: Introduce feature flag (disabled by default)
- Add `@aws-cdk/core:versionedCustomResourceHandlers` flag
- Implement content-based hashing logic
- Update custom resource handler creation

**Phase 2**: Enable for new projects
- Add to recommended feature flags
- Update documentation and examples

**Phase 3**: Make default in CDK v3
- Enable by default for all new stacks
- Provide migration guide for existing stacks

**Phase 4**: Deprecate old behavior
- Remove non-versioned handler support in future major version

## Affected Custom Resources

This change would benefit all CDK custom resources, including but not limited to:

- S3 Bucket Deployment
- ECR Image Asset
- Lambda Layer Version
- Route53 Cross-Account Zone Delegation
- CloudFront Distribution Invalidation
- DynamoDB Table Replication
- ECS Service Discovery
- And many more...

## Alternative Approaches Considered

### 1. Explicit Version Pinning
Allow users to specify handler versions in `cdk.json`:
```json
{
"@aws-cdk/core:customResourceHandlerVersion": "v2.100.0"
}
```
**Rejected**: Requires manual user intervention and doesn't provide automatic safety

### 2. Lambda Layers
Package handlers as versioned Lambda layers
**Rejected**: Layer size limits and added complexity

### 3. Immutable S3 Paths
Deploy handlers to versioned S3 paths based on CDK version
**Rejected**: Ties handler versions to CDK versions, not handler code changes

## Open Questions

1. Should there be a mechanism to force upgrade all stacks to the latest handler version?
2. How should we handle cleanup of very old handler versions (e.g., 10+ versions old)?
3. Should the hash be based on code content only, or include runtime/dependencies?
4. Should this apply to all custom resources or be opt-in per resource type?

## Success Metrics

- Zero unintended handler updates in existing deployments
- Reduced bug reports related to custom resource handler changes
- Increased confidence in CDK upgrades
- Positive community feedback on stability

## References

- Related to the general problem of CDK construct versioning
- Similar to how Lambda function versions and aliases work
- Aligns with immutable infrastructure principles

---

Start by locating the custom resource handler creation and the SingletonFunction usage described in the proposal, then trace how handlerPath and the feature-flag configuration are consumed. Done means content-based handler versions, opt-in flag behavior, backward compatibility, and coverage for affected custom resources are defined and verified.

Written by the indexing model from the issue text.