aws / aws/aws-cdk

(custom-resources): `AwsCustomResource` can fail during stack update when a new instance adds permissions to the shared singleton provider role

Open
#37,675 2 comments 0 reactions 0 assignees View on GitHub
@aws-cdk/aws-iam @aws-cdk/custom-resources aws-cdk-lib bug p2
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

I hit what looks like an undocumented, deployment-time sharp edge in [`AwsCustomResource`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.custom_resources.AwsCustomResource.html).

When multiple `AwsCustomResource` instances in the same stack use the default singleton provider Lambda and each instance contributes its own `policy`, adding a new instance during a stack update can intermittently fail with `AccessDenied` even though the synthesized template contains the correct new `AWS::IAM::Policy` and the custom resource depends on that policy.

The docs explain that `AwsCustomResource` uses a singleton provider Lambda and that its role accumulates permissions from all `AwsCustomResource` instances in the stack. What seems to be missing is a warning that adding new permissions to that shared role during a stack update can be timing-sensitive.

## Example Pattern

I have a wrapper construct that does a deployment-time lookup using `AwsCustomResource`. There are several instances of this pattern in the same stack, each targeting a different ECR repository.

```ts
import { AwsCustomResource, AwsCustomResourcePolicy, PhysicalResourceId } from "aws-cdk-lib/custom-resources";
import { PolicyStatement } from "aws-cdk-lib/aws-iam";

new AwsCustomResource(this, "Lookup", {
onUpdate: {
service: "ECR",
action: "describeImages",
parameters: {
registryId: "",
repositoryName: "example/example-repo-b",
imageIds: [{ imageTag: "v1.2.3" }],
},
region: "",
physicalResourceId: PhysicalResourceId.fromResponse("imageDetails.0.imageDigest"),
},
policy: AwsCustomResourcePolicy.fromStatements([
new PolicyStatement({
actions: ["ecr:DescribeImages"],
resources: ["arn:aws:ecr:::repository/example/example-repo-b"],
}),
]),
installLatestAwsSdk: false,
});
```

There are multiple instances like this in the same stack:

- one for `example_repo_a`
- one for `example_repo_b`
- one for `example_repo_c`

The failure appeared when `example-repo-b` was newly added in an update.

## What Happened

During the stack update:

1. Existing custom resource lookups succeeded.
2. The newly added lookup failed with `AccessDenied`.
3. The error said the singleton provider role was not authorized for the new resource.
4. The synthesized template already contained the corresponding new `AWS::IAM::Policy`.

I then reran the exact same workflow without any code changes, and the deployment succeeded.

On the successful rerun, CloudFormation events showed the expected ordering for the newly added lookup:

1. the generated `AWS::IAM::Policy` for the new custom resource reached `CREATE_COMPLETE`
2. only after that, the corresponding `Custom::AWS` resource started
3. the custom resource then completed successfully

That makes this look less like a missing dependency edge in the synthesized template and more like an intermittent IAM consistency / permission visibility issue affecting the shared singleton provider role.

Example runtime error:

```text
Received response status [FAILED] from custom resource.
Message returned: User: arn:aws:sts:::assumed-role/-/...
is not authorized to perform: ecr:DescribeImages on resource:
arn:aws:ecr:::repository/example/example-repo-b
because no identity-based policy allows the ecr:DescribeImages action
```

## Why This Is Surprising

The generated template had the expected structure:

- one `Custom::AWS` resource per logical lookup
- one generated provider Lambda shared across all of them
- one generated provider role shared across all of them
- one generated `AWS::IAM::Policy` per lookup, each attached to the same shared role

Each custom resource depended on its own generated policy resource, so from the synthesized template it looked like the needed policy should exist before the corresponding invocation.

The successful rerun reinforced that interpretation: the CloudFormation event stream showed the policy reaching `CREATE_COMPLETE` before the custom resource started, yet the earlier run still failed.

## Architecture

```mermaid
flowchart LR
A["AwsCustomResource A\nlookup repo-a"] -->|ServiceToken| P["Shared singleton provider Lambda"]
B["AwsCustomResource B\nlookup repo-b"] -->|ServiceToken| P
C["AwsCustomResource C\nlookup repo-c"] -->|ServiceToken| P

P -->|uses| R["Shared provider IAM role"]

PA["Policy A\nrepo-a"] --> R
PB["Policy B\nrepo-b"] --> R
PC["Policy C\nrepo-c"] --> R
```

## Dependency Shape

The important detail is that each custom resource depends on its own generated policy resource, but not on all sibling policies.

```mermaid
flowchart TD
Role["Shared provider role"]
PolicyA["Policy A"] --> Role
PolicyB["Policy B"] --> Role
PolicyC["Policy C"] --> Role

CRA["Custom resource A"] -->|DependsOn| PolicyA
CRB["Custom resource B"] -->|DependsOn| PolicyB
CRC["Custom resource C"] -->|DependsOn| PolicyC

CRA --> Provider["Shared provider Lambda"]
CRB --> Provider
CRC --> Provider
```

That means the provider role is effectively being extended incrementally while the provider is being used.

## Likely Issue

I want to be careful not to overstate the exact AWS-internal mechanism.

I cannot prove from the logs alone whether this is specifically caused by cached STS credentials, Lambda execution environment reuse, IAM propagation delay, or some combination of those.

What I can say confidently is:

- the required new policy was present in the synthesized template
- the custom resource depended on that policy resource
- the provider still received `AccessDenied` for the newly introduced permission during the same stack update
- rerunning the same deployment without changes succeeded
- on the successful rerun, the CloudFormation events showed the policy reaching `CREATE_COMPLETE` before the custom resource invocation

So the effective permissions visible to the shared provider role were incomplete at invocation time during at least one run, despite correct synthesis and apparently correct resource ordering.

## Why This Feels Like A CDK Issue

From the consumer API, `policy` is specified per `AwsCustomResource` instance, which suggests each instance is fairly self-contained.

In reality:

- all instances share one provider Lambda
- all instances share one provider role
- all per-instance `policy` fragments are accumulated onto that one role

That makes per-instance `policy` updates timing-sensitive in a way that is not obvious from the construct API.

At minimum this seems under-documented. It may also be a design issue in the default singleton provider model when used with evolving per-instance permissions.

## Suggested Improvements

Any of these would help:

1. Documentation warning that per-instance `policy` values are accumulated onto a shared singleton provider role and may be unsafe when new permissions are introduced during the same deployment.
2. Guidance to prefer an explicit shared `role` when many `AwsCustomResource` instances are expected to evolve over time.
3. Potential guardrails or warnings when multiple instances attach distinct narrow policies to the default singleton role.

Contributor guide

Open the contributing guide

Research direction

Start at the AwsCustomResource construct and its default singleton provider and shared role behavior; reproduce an update that adds a new instance with a distinct policy and inspect the synthesized dependencies and CloudFormation events. Done means the intermittent AccessDenied behavior is explained and addressed, or the shared-role limitation is documented with clear guidance.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
cloud, infrastructure, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.