aws / aws/aws-cdk

(core): docker login to deployment account ECR occurs before asset is built

Open
#25,894 10 comments 11 reactions 0 assignees View on GitHub
@aws-cdk/core bug documentation effort/small p2
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

### Describe the bug

Given a simple `Dockerfile` that pulls from a private ECR repository in the same account you're deploying to:

```docker
ARG AWS_ACCOUNT_NUMBER
ARG AWS_REGION
ARG REPO
ARG TAG

FROM ${AWS_ACCOUNT_NUMBER}.dkr.ecr.${AWS_REGION}.amazonaws.com/${REPO}:${TAG}
```

With a `DockerImageAsset`:

```ts
import * as cdk from 'aws-cdk-lib';
import { DockerImageAsset } from 'aws-cdk-lib/aws-ecr-assets';
import { Construct } from 'constructs';
import { join } from 'path';

export class CdkBugReportsStack extends cdk.Stack {
constructor(scope: Construct, id: string, props?: cdk.StackProps) {
super(scope, id, props);

new DockerImageAsset(this, 'DockerImageAsset', {
directory: join(__dirname, '..', 'assets', 'docker'),
buildArgs: {
"AWS_ACCOUNT_NUMBER": "123456789012",
"AWS_REGION": "us-west-2",
"REPO": "my-repo",
"TAG": "latest"
}
})
}
}
```

The `cdk deploy` will fail with a message that looks like this:

```
#3 [internal] load metadata for .dkr.ecr..amazonaws.com/:
#3 ERROR: pulling from host.dkr.ecr..amazonaws.com failed with status code [manifests ]: 403 Forbidden
```

So, in other words, the `FROM` in the `Dockerfile` cannot be resolved. The reason this happens is because the image publishing role (`arn:aws:iam:::role/cdk-hnb659fds-image-publishing-role--`) is used to login to docker before the image is built.

Therefore, it overrides the existing `docker login` you might have already done via:

```
> aws ecr get-login-password --region us-west-2 | docker login --username AWS --password-stdin .dkr.ecr..amazonaws.com
```

And then it can't pull using existing credentials you've already set up.

### Expected Behavior

I expected the system-level `docker login` to be respected during image build time so it could resolve the image from the private ECR repo.

I understand that the `image-publishing-role` needs to be assumed to push to the CDK assets ECR repository, but it feels like those credentials should only be used before calling `docker push`.

In other words, the flow looks like:

1. Standard `docker login` happens as a setup step in my CI platform
2. `DockerImageAsset` is built using the credentials from step 1.
3. Existing `docker login` is backed up
4. `docker login` occurs for `image-publishing-role`
5. `docker push` the built asset
6. Restore saved `docker login` credentials from step 3

### Current Behavior

What's happening now appears to be:

1. Standard `docker login` happens as a setup step in my CI platform
2. `docker login` occurs for `image-publishing-role`
3. `DockerImageAsset` is built using the credentials from step 2 (failure because `image-publishing-role` can't access the private ECR repo.

### Reproduction Steps

https://github.com/blimmer/cdk-bug-reports/pull/2 shows an example. You do need to manually push a `latest` tag to the repo to make it technically correct. However, you should still see the error even with an empty repo (you'll get a 403 error).

### Possible Solution

If possible, system `docker login`s should be used to build the Docker images, not the `image-publishing-role`.

It might be challenging, however, to back up docker credentials, since there are a few different ways you can store those values.

### Additional Information/Context

You can work around this issue by applying a policy to your private repository that allows the `image-publishing-role` access to the repo.

### CDK CLI Version

2.83.0

### Framework Version

_No response_

### Node.js Version

18

### OS

MacOS

### Language

Typescript

### Language Version

_No response_

### Other information

_No response_

Contributor guide

Open the contributing guide

Research direction

Start from the CDK CLI flow that handles DockerImageAsset builds and the image-publishing role; the issue does not name specific source files or tests. Reproduce the failure with the linked example and the private ECR Dockerfile, then verify that existing Docker credentials are used during the build while publishing-role credentials are used only for the asset push.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, docker, typescript
Domain
cloud, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.