aws / aws/aws-cdk

(msk): associate secret with cluster

Open
#20,490 8 comments 0 reactions 0 assignees View on GitHub
@aws-cdk/aws-msk effort/small feature-request p2
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

### Describe the bug

I have 2 stacks

mskStack and opensearchStack

mskStack creates secret containing user credentials for my mskCluster

In kafkaStack:
```
kafkaCluster.addUser("KafkaUsername");
```

The secret created in the mskStack is needed again in my opensearchStack as I have defined a lambda function to use mskEventListener to send data from msk to my opensearch domain.

Due to the secret being created behind the scenes by the mskCluster construct, I do not have a secret defined in my mskStack to output, so the secret must be referenced as an existing resource. I tried to accomplish this via the following implementation:

In opensearchStack:
```
const user = secretsmanager.Secret.fromSecretNameV2(this, "kafkaClusterUser", "AmazonMSK_KafkaUsername")

mskLambdaConsumer.addEventSource(new events.ManagedKafkaEventSource({
batchSize: 100,
clusterArn: props.kafkaClusterArn,
maxBatchingWindow: cdk.Duration.seconds(10),
secret: user,
startingPosition: lambda.StartingPosition.TRIM_HORIZON,
topic: "topicName",
}));

```

The above code fails, however, if I remove the `secret: user,` attribute line from the `mskLambdaConsumer.addEventSource` definition, then the mskEventSource creates without a problem.

Furthermore, if I manually create the EventSource in the AWS console, and select the very same secret to use for the mskEventSource Authentication option, the mskEventSource successfully links to the secret and can access the mskCluster.

This establishes the following:
- the secret is valid
- the mskCluster is valid
- the mskEventSource is valid

The breakdown is when cdk uses `fromSecretNameV2` to retrieve the secret and then pass the retrieved value to the mskEventSource

### Expected Behavior

A valid secret to be retrieved from secretsManager

### Current Behavior

OpenSearchStack failed:
Error: The stack named OpenSearchStack failed to deploy:
UPDATE_ROLLBACK_COMPLETE: Resource handler returned message:
"Invalid request provided: The secret provided in 'sourceAccessConfigurations' is not associated with cluster . Please provide a secret associated with the cluster. (Service: Lambda, Status Code: 400, Request ID: , Extended Request ID: null)" (RequestToken: , HandlerErrorCode: InvalidRequest), Resource handler returned message: "Invalid request provided: The secret provided in 'sourceAccessConfigurations' is not associated with cluster . Please provide a secret associated with the cluster. (Service: Lambda, Status Code: 400, Request ID: , Extended Request ID: null)" (RequestToken: 695cbaba-3861-87ad-7c33-8394ea84f506, HandlerErrorCode: InvalidRequest)
at prepareAndExecuteChangeSet (/PATH/node_modules/aws-cdk/lib/api/deploy-stack.ts:385:13)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at CdkToolkit.deploy (/PATH/node_modules/aws-cdk/lib/cdk-toolkit.ts:209:24)
at initCommandLine (/PATH/node_modules/aws-cdk/lib/cli.ts:341:12)

The stack named OpenSearchStack failed to deploy: UPDATE_ROLLBACK_COMPLETE: Resource handler returned message: "Invalid request provided: The secret provided in 'sourceAccessConfigurations' is not associated with cluster . Please provide a secret associated with the cluster. (Service: Lambda, Status Code: 400, Request ID: , Extended Request ID: null)" (RequestToken: , HandlerErrorCode: InvalidRequest), Resource handler returned message: "Invalid request provided: The secret provided in 'sourceAccessConfigurations' is not associated with cluster . Please provide a secret associated with the cluster. (Service: Lambda, Status Code: 400, Request ID: , Extended Request ID: null)" (RequestToken: 695cbaba-3861-87ad-7c33-8394ea84f506, HandlerErrorCode: InvalidRequest)

### Reproduction Steps

```
const user = secretsmanager.Secret.fromSecretNameV2(this, "kafkaClusterUser", )
.addEventSource(new events.ManagedKafkaEventSource({
batchSize: 100,
clusterArn: ,
maxBatchingWindow: cdk.Duration.seconds(10),
secret: user,
startingPosition: lambda.StartingPosition.TRIM_HORIZON,
topic: ,
}));
```

### Possible Solution

_No response_

### Additional Information/Context

_No response_

### CDK CLI Version

2.2.0

### Framework Version

2.25.0

### Node.js Version

7.15.1

### OS

OSX Monterey, 12.4

### Language

Typescript

### Language Version

~3.9.7

### Other information

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the ManagedKafkaEventSource configuration and the Secret.fromSecretNameV2 call shown in the reproduction, then compare the generated event-source configuration with the working AWS console setup. Done means the imported secret is accepted for the existing MSK cluster and the stack deploys successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.