(msk): associate secret with cluster
- Dominant language
- TypeScript
- Stars
- 12.9k
- Forks
- 4.6k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 83
Description
### Describe the bug
I have 2 stacks
mskStack and opensearchStack
mskStack creates secret containing user credentials for my mskCluster
In kafkaStack:
```
kafkaCluster.addUser("KafkaUsername");
```
The secret created in the mskStack is needed again in my opensearchStack as I have defined a lambda function to use mskEventListener to send data from msk to my opensearch domain.
Due to the secret being created behind the scenes by the mskCluster construct, I do not have a secret defined in my mskStack to output, so the secret must be referenced as an existing resource. I tried to accomplish this via the following implementation:
In opensearchStack:
```
const user = secretsmanager.Secret.fromSecretNameV2(this, "kafkaClusterUser", "AmazonMSK_KafkaUsername")
mskLambdaConsumer.addEventSource(new events.ManagedKafkaEventSource({
batchSize: 100,
clusterArn: props.kafkaClusterArn,
maxBatchingWindow: cdk.Duration.seconds(10),
secret: user,
startingPosition: lambda.StartingPosition.TRIM_HORIZON,
topic: "topicName",
}));
```
The above code fails, however, if I remove the `secret: user,` attribute line from the `mskLambdaConsumer.addEventSource` definition, then the mskEventSource creates without a problem.
Furthermore, if I manually create the EventSource in the AWS console, and select the very same secret to use for the mskEventSource Authentication option, the mskEventSource successfully links to the secret and can access the mskCluster.
This establishes the following:
- the secret is valid
- the mskCluster is valid
- the mskEventSource is valid
The breakdown is when cdk uses `fromSecretNameV2` to retrieve the secret and then pass the retrieved value to the mskEventSource
### Expected Behavior
A valid secret to be retrieved from secretsManager
### Current Behavior
OpenSearchStack failed:
Error: The stack named OpenSearchStack failed to deploy:
UPDATE_ROLLBACK_COMPLETE: Resource handler returned message:
"Invalid request provided: The secret provided in 'sourceAccessConfigurations' is not associated with cluster . Please provide a secret associated with the cluster. (Service: Lambda, Status Code: 400, Request ID: , Extended Request ID: null)" (RequestToken: , HandlerErrorCode: InvalidRequest), Resource handler returned message: "Invalid request provided: The secret provided in 'sourceAccessConfigurations' is not associated with cluster . Please provide a secret associated with the cluster. (Service: Lambda, Status Code: 400, Request ID: , Extended Request ID: null)" (RequestToken: 695cbaba-3861-87ad-7c33-8394ea84f506, HandlerErrorCode: InvalidRequest)
at prepareAndExecuteChangeSet (/PATH/node_modules/aws-cdk/lib/api/deploy-stack.ts:385:13)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at CdkToolkit.deploy (/PATH/node_modules/aws-cdk/lib/cdk-toolkit.ts:209:24)
at initCommandLine (/PATH/node_modules/aws-cdk/lib/cli.ts:341:12)
The stack named OpenSearchStack failed to deploy: UPDATE_ROLLBACK_COMPLETE: Resource handler returned message: "Invalid request provided: The secret provided in 'sourceAccessConfigurations' is not associated with cluster . Please provide a secret associated with the cluster. (Service: Lambda, Status Code: 400, Request ID: , Extended Request ID: null)" (RequestToken: , HandlerErrorCode: InvalidRequest), Resource handler returned message: "Invalid request provided: The secret provided in 'sourceAccessConfigurations' is not associated with cluster . Please provide a secret associated with the cluster. (Service: Lambda, Status Code: 400, Request ID: , Extended Request ID: null)" (RequestToken: 695cbaba-3861-87ad-7c33-8394ea84f506, HandlerErrorCode: InvalidRequest)
### Reproduction Steps
```
const user = secretsmanager.Secret.fromSecretNameV2(this, "kafkaClusterUser", )
.addEventSource(new events.ManagedKafkaEventSource({
batchSize: 100,
clusterArn: ,
maxBatchingWindow: cdk.Duration.seconds(10),
secret: user,
startingPosition: lambda.StartingPosition.TRIM_HORIZON,
topic: ,
}));
```
### Possible Solution
_No response_
### Additional Information/Context
_No response_
### CDK CLI Version
2.2.0
### Framework Version
2.25.0
### Node.js Version
7.15.1
### OS
OSX Monterey, 12.4
### Language
Typescript
### Language Version
~3.9.7
### Other information
_No response_
Contributor guide
Research direction
Start with the ManagedKafkaEventSource configuration and the Secret.fromSecretNameV2 call shown in the reproduction, then compare the generated event-source configuration with the working AWS console setup. Done means the imported secret is accepted for the existing MSK cluster and the stack deploys successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, typescript
- Domain
- cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100