aws / aws/aws-cdk

aws-opensearchservice: AccessDenied on Custom::OpenSearchAccessPolicy due to es:UpdateDomainConfig IAM propagation race

Open
#37,939 1 comment 0 reactions 0 assignees View on GitHub
@aws-cdk/aws-opensearch bug p2
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
1d 19h
Merged PRs (30d)
71

Description

## Describe the bug

The `aws-opensearchservice.Domain` L2 construct applies `accessPolicies` indirectly via a Lambda-backed `Custom::OpenSearchAccessPolicy` resource that calls `UpdateDomainConfig` after the domain reaches `CREATE_COMPLETE`. The custom resource's IAM execution role is permissioned at deploy time, and the `es:UpdateDomainConfig` grant has been observed to take longer to propagate than the `CREATE_COMPLETE` callback. This produces intermittent `AccessDenied` failures, especially on stacks that create several managed domains in parallel.

## Expected Behavior

`accessPolicies` should be applied to the domain reliably without depending on a post-create custom-resource callback whose IAM grant may not have propagated.

## Current Behavior

`AWS::CloudFormation` errors during `CREATE_IN_PROGRESS` of `Custom::OpenSearchAccessPolicy`, with a Lambda log line resembling:

```
AccessDenied: User: arn:aws:sts:::assumed-role/-AwsCdkSdkCallProvider...-...-... is not authorized to perform: es:UpdateDomainConfig on resource: arn:aws:es:::domain/ because no identity-based policy allows the es:UpdateDomainConfig action
```

The failure is intermittent — re-running the same stack often succeeds. Failure rate scales with the number of managed domains created in parallel within a single deployment.

Observed in the wild on [`opensearch-project/opensearch-migrations`](https://github.com/opensearch-project/opensearch-migrations) where stacks routinely provision multiple OpenSearch domains side-by-side.

## Reproduction Steps

```ts
import { App, Stack } from 'aws-cdk-lib';
import * as opensearch from 'aws-cdk-lib/aws-opensearchservice';
import * as iam from 'aws-cdk-lib/aws-iam';

const app = new App();
const stack = new Stack(app, 'Repro');

// Create several domains in parallel that all use the custom-resource path.
for (let i = 0; i < 4; i++) {
new opensearch.Domain(stack, `D${i}`, {
version: opensearch.EngineVersion.OPENSEARCH_2_17,
domainName: `repro-${i}`,
accessPolicies: [
new iam.PolicyStatement({
effect: iam.Effect.ALLOW,
principals: [new iam.AccountRootPrincipal()],
actions: ['es:ESHttp*'],
resources: [`arn:aws:es:us-east-1:111111111111:domain/repro-${i}/*`],
}),
],
});
}
```

Deploy. Some `Custom::OpenSearchAccessPolicy` resources will fail with `AccessDenied: es:UpdateDomainConfig` while others succeed — a textbook IAM-propagation race.

## Possible Solution

`AWS::OpenSearchService::Domain` natively supports an inline `AccessPolicies` property. Writing the policy directly onto the L1 resource eliminates the post-create custom-resource hop entirely (no Lambda role, no `es:UpdateDomainConfig` grant required, no IAM-propagation race).

The reason the construct historically chose the custom-resource path is the `Fn::GetAtt` self-cycle that arises when `accessPolicies` references the domain's own ARN (e.g. `${domain.domainArn}/*`, used internally by `useUnsignedBasicAuth`). CloudFormation rejects self-references on `AccessPolicies`.

This can be resolved by introducing a feature flag — `@aws-cdk/aws-opensearchservice:inlineAccessPolicies` — that:

1. Writes `accessPolicies` directly onto the L1 resource when safe.
2. Detects self-references at synth time. When the domain has a synth-time-resolvable `domainName`, rewrites every `Fn::GetAtt` / `Ref` to the domain's own logical id into a literal ARN built via `Stack.formatArn`. This keeps `useUnsignedBasicAuth` and self-referential user policies on the inline path.
3. Falls back to today's custom-resource behavior only when CloudFormation owns the physical name (i.e. `domainName` was not provided).

This is what #37918 implements.

## Additional Information/Context

- Related to internal IAM eventual-consistency behavior; unlikely to be addressable on the OpenSearch service side because the failure is in the construct's own deploy-time custom resource, not in OpenSearch.
- The fallback path preserves today's behavior bit-for-bit for users who don't opt into the flag (or for users whose domains lack a `domainName`).
- A linked PR (#37918) ships the fix and includes:
- Unit tests covering both the inline-with-rewrite path and the custom-resource-fallback path.
- An integration test that deploys a real domain with `${domain.domainArn}/*` in `accessPolicies` and asserts via `OpenSearch::DescribeDomainConfig` that the inline policy was applied at create time.

## AWS CDK Library version (aws-cdk-lib)

2.255.0 (and earlier; this is long-standing behavior of the L2 `Domain` construct)

## AWS CDK CLI version

2.1018.0+

## Node.js Version

20.x / 22.x (any current LTS)

## OS

Any

## Language

TypeScript

Contributor guide

Open the contributing guide

Research direction

Start with the aws-opensearchservice.Domain L2 construct and the behavior described for Custom::OpenSearchAccessPolicy. Review PR #37918 and its unit tests for the inline-with-rewrite and custom-resource-fallback paths, plus the integration test using ${domain.domainArn}/*. Done means access policies avoid the IAM propagation race while fallback behavior remains covered.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.