aws-opensearchservice: AccessDenied on Custom::OpenSearchAccessPolicy due to es:UpdateDomainConfig IAM propagation race
- Dominant language
- TypeScript
- Stars
- 12.9k
- Forks
- 4.6k
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 71
Description
## Describe the bug
The `aws-opensearchservice.Domain` L2 construct applies `accessPolicies` indirectly via a Lambda-backed `Custom::OpenSearchAccessPolicy` resource that calls `UpdateDomainConfig` after the domain reaches `CREATE_COMPLETE`. The custom resource's IAM execution role is permissioned at deploy time, and the `es:UpdateDomainConfig` grant has been observed to take longer to propagate than the `CREATE_COMPLETE` callback. This produces intermittent `AccessDenied` failures, especially on stacks that create several managed domains in parallel.
## Expected Behavior
`accessPolicies` should be applied to the domain reliably without depending on a post-create custom-resource callback whose IAM grant may not have propagated.
## Current Behavior
`AWS::CloudFormation` errors during `CREATE_IN_PROGRESS` of `Custom::OpenSearchAccessPolicy`, with a Lambda log line resembling:
```
AccessDenied: User: arn:aws:sts:::assumed-role/-AwsCdkSdkCallProvider...-...-... is not authorized to perform: es:UpdateDomainConfig on resource: arn:aws:es:::domain/ because no identity-based policy allows the es:UpdateDomainConfig action
```
The failure is intermittent — re-running the same stack often succeeds. Failure rate scales with the number of managed domains created in parallel within a single deployment.
Observed in the wild on [`opensearch-project/opensearch-migrations`](https://github.com/opensearch-project/opensearch-migrations) where stacks routinely provision multiple OpenSearch domains side-by-side.
## Reproduction Steps
```ts
import { App, Stack } from 'aws-cdk-lib';
import * as opensearch from 'aws-cdk-lib/aws-opensearchservice';
import * as iam from 'aws-cdk-lib/aws-iam';
const app = new App();
const stack = new Stack(app, 'Repro');
// Create several domains in parallel that all use the custom-resource path.
for (let i = 0; i < 4; i++) {
new opensearch.Domain(stack, `D${i}`, {
version: opensearch.EngineVersion.OPENSEARCH_2_17,
domainName: `repro-${i}`,
accessPolicies: [
new iam.PolicyStatement({
effect: iam.Effect.ALLOW,
principals: [new iam.AccountRootPrincipal()],
actions: ['es:ESHttp*'],
resources: [`arn:aws:es:us-east-1:111111111111:domain/repro-${i}/*`],
}),
],
});
}
```
Deploy. Some `Custom::OpenSearchAccessPolicy` resources will fail with `AccessDenied: es:UpdateDomainConfig` while others succeed — a textbook IAM-propagation race.
## Possible Solution
`AWS::OpenSearchService::Domain` natively supports an inline `AccessPolicies` property. Writing the policy directly onto the L1 resource eliminates the post-create custom-resource hop entirely (no Lambda role, no `es:UpdateDomainConfig` grant required, no IAM-propagation race).
The reason the construct historically chose the custom-resource path is the `Fn::GetAtt` self-cycle that arises when `accessPolicies` references the domain's own ARN (e.g. `${domain.domainArn}/*`, used internally by `useUnsignedBasicAuth`). CloudFormation rejects self-references on `AccessPolicies`.
This can be resolved by introducing a feature flag — `@aws-cdk/aws-opensearchservice:inlineAccessPolicies` — that:
1. Writes `accessPolicies` directly onto the L1 resource when safe.
2. Detects self-references at synth time. When the domain has a synth-time-resolvable `domainName`, rewrites every `Fn::GetAtt` / `Ref` to the domain's own logical id into a literal ARN built via `Stack.formatArn`. This keeps `useUnsignedBasicAuth` and self-referential user policies on the inline path.
3. Falls back to today's custom-resource behavior only when CloudFormation owns the physical name (i.e. `domainName` was not provided).
This is what #37918 implements.
## Additional Information/Context
- Related to internal IAM eventual-consistency behavior; unlikely to be addressable on the OpenSearch service side because the failure is in the construct's own deploy-time custom resource, not in OpenSearch.
- The fallback path preserves today's behavior bit-for-bit for users who don't opt into the flag (or for users whose domains lack a `domainName`).
- A linked PR (#37918) ships the fix and includes:
- Unit tests covering both the inline-with-rewrite path and the custom-resource-fallback path.
- An integration test that deploys a real domain with `${domain.domainArn}/*` in `accessPolicies` and asserts via `OpenSearch::DescribeDomainConfig` that the inline policy was applied at create time.
## AWS CDK Library version (aws-cdk-lib)
2.255.0 (and earlier; this is long-standing behavior of the L2 `Domain` construct)
## AWS CDK CLI version
2.1018.0+
## Node.js Version
20.x / 22.x (any current LTS)
## OS
Any
## Language
TypeScript
Contributor guide
Research direction
Start with the aws-opensearchservice.Domain L2 construct and the behavior described for Custom::OpenSearchAccessPolicy. Review PR #37918 and its unit tests for the inline-with-rewrite and custom-resource-fallback paths, plus the integration test using ${domain.domainArn}/*. Done means access policies avoid the IAM propagation race while fallback behavior remains covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, typescript
- Domain
- cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 25/100