aws-msk: scram secret association fails with ErrorCode InvalidSecretArn
- Dominant language
- TypeScript
- Stars
- 12.9k
- Forks
- 4.6k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 83
Description
### Describe the bug
Adding new users to our MSK cluster is relatively not often happening, but have noticed that sometimes it fails associating secret with the cluster.
There are two problems here: firstly error validation is not done correctly. Cloudformation reports that everything is ok and I can only learn that it is not OK after long research. This is very detrimental for DX of AWS deployments. The whole reason why we use cloudformation in the first place with its inherent complexity is that it is supposed to be deeply integrated with AWS and that includes giving correct responses when something goes wrong.
Secondly is the problem with creation itself that looks like this in Cloudwatch logs:
```
API response {
ClusterArn: 'arn:aws:kafka:eu-central-1::cluster/KafkaCluster/',
'UnprocessedScramSecrets.0.ErrorCode': 'InvalidSecretArn',
'UnprocessedScramSecrets.0.ErrorMessage': "Amazon MSK wasn't able to access the provided secret.",
'UnprocessedScramSecrets.0.SecretArn': 'arn:aws:secretsmanager:eu-central-1::secret:AmazonMSK_KafkaCluster_my-service-zKtqpf'
}
```
Directly after this log is the following:
```
Responding {
"Status": "SUCCESS",
"Reason": "OK",
"PhysicalResourceId": "CreateUsers",
...
```
, which shows that the error was not correctly passed on to cloudformation (it reports "SUCCESS", even though it clearly failed).
I solve this usually by removing user from CDK and then adding it again (which is another confirmation that the bug is in aws-cdk).
I have scanned CDK logs for as war as they go and found 3 total failures like that for maybe total 20-30 executions of that association logic in two different accounts, so it does not happen every time, but often enough.
### Regression Issue
- [ ] Select this option if this issue appears to be a regression.
### Last Known Working CDK Library Version
_No response_
### Expected Behavior
if error happened - cloudformation stack should show it
and preferably this error should not happen
### Current Behavior
when error happens, stack shows successful update, but secret is not associated and my service cannot connect to Kafka
### Reproduction Steps
Something like this should do it:
```typescript
const mycluster = new msk.Cluster( this, "Cluster", {
clusterName : "myCluster",
kafkaVersion : msk.KafkaVersion.V3_4_0,
vpc,
clientAuthentication : msk.ClientAuthentication.sasl( {
scram : true,
} ),
} );
mycluster.addUser( "my-service" );
```
Just run it 20-30 times (with destroying and recreating every time), it should probably be triggered once or twice.
### Possible Solution
First of all lambda that processes the custom resource lifecycle should report failure correctly if it could not do association.
Secondly the reason for why arn is reported incorrect should be investigated within secrets manager or whatever returned that error code (AWS SDK maybe?). ARN was correct, so validation in secret manager or its SDK must be wrong somehow.
### Additional Information/Context
_No response_
### AWS CDK Library version (aws-cdk-lib)
2.192.0
### AWS CDK CLI version
2.1012.0
### Node.js Version
v20.18.0
### OS
Ubuntu 24.04
### Language
TypeScript
### Language Version
_No response_
### Other information
_No response_
Contributor guide
Research direction
Start with the TypeScript reproduction using msk.Cluster and mycluster.addUser, then trace the custom resource lifecycle that associates the SCRAM secret. Reproduce the intermittent InvalidSecretArn response by repeating deployment and destruction, and verify that association errors are reported as CloudFormation failures rather than SUCCESS; investigate the ARN failure separately if it remains reproducible.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, typescript
- Domain
- cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100