(core): docker login to deployment account ECR occurs before asset is built
- Dominant language
- TypeScript
- Stars
- 12.9k
- Forks
- 4.6k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 83
Description
### Describe the bug
Given a simple `Dockerfile` that pulls from a private ECR repository in the same account you're deploying to:
```docker
ARG AWS_ACCOUNT_NUMBER
ARG AWS_REGION
ARG REPO
ARG TAG
FROM ${AWS_ACCOUNT_NUMBER}.dkr.ecr.${AWS_REGION}.amazonaws.com/${REPO}:${TAG}
```
With a `DockerImageAsset`:
```ts
import * as cdk from 'aws-cdk-lib';
import { DockerImageAsset } from 'aws-cdk-lib/aws-ecr-assets';
import { Construct } from 'constructs';
import { join } from 'path';
export class CdkBugReportsStack extends cdk.Stack {
constructor(scope: Construct, id: string, props?: cdk.StackProps) {
super(scope, id, props);
new DockerImageAsset(this, 'DockerImageAsset', {
directory: join(__dirname, '..', 'assets', 'docker'),
buildArgs: {
"AWS_ACCOUNT_NUMBER": "123456789012",
"AWS_REGION": "us-west-2",
"REPO": "my-repo",
"TAG": "latest"
}
})
}
}
```
The `cdk deploy` will fail with a message that looks like this:
```
#3 [internal] load metadata for .dkr.ecr..amazonaws.com/:
#3 ERROR: pulling from host.dkr.ecr..amazonaws.com failed with status code [manifests ]: 403 Forbidden
```
So, in other words, the `FROM` in the `Dockerfile` cannot be resolved. The reason this happens is because the image publishing role (`arn:aws:iam:::role/cdk-hnb659fds-image-publishing-role--`) is used to login to docker before the image is built.
Therefore, it overrides the existing `docker login` you might have already done via:
```
> aws ecr get-login-password --region us-west-2 | docker login --username AWS --password-stdin .dkr.ecr..amazonaws.com
```
And then it can't pull using existing credentials you've already set up.
### Expected Behavior
I expected the system-level `docker login` to be respected during image build time so it could resolve the image from the private ECR repo.
I understand that the `image-publishing-role` needs to be assumed to push to the CDK assets ECR repository, but it feels like those credentials should only be used before calling `docker push`.
In other words, the flow looks like:
1. Standard `docker login` happens as a setup step in my CI platform
2. `DockerImageAsset` is built using the credentials from step 1.
3. Existing `docker login` is backed up
4. `docker login` occurs for `image-publishing-role`
5. `docker push` the built asset
6. Restore saved `docker login` credentials from step 3
### Current Behavior
What's happening now appears to be:
1. Standard `docker login` happens as a setup step in my CI platform
2. `docker login` occurs for `image-publishing-role`
3. `DockerImageAsset` is built using the credentials from step 2 (failure because `image-publishing-role` can't access the private ECR repo.
### Reproduction Steps
https://github.com/blimmer/cdk-bug-reports/pull/2 shows an example. You do need to manually push a `latest` tag to the repo to make it technically correct. However, you should still see the error even with an empty repo (you'll get a 403 error).
### Possible Solution
If possible, system `docker login`s should be used to build the Docker images, not the `image-publishing-role`.
It might be challenging, however, to back up docker credentials, since there are a few different ways you can store those values.
### Additional Information/Context
You can work around this issue by applying a policy to your private repository that allows the `image-publishing-role` access to the repo.
### CDK CLI Version
2.83.0
### Framework Version
_No response_
### Node.js Version
18
### OS
MacOS
### Language
Typescript
### Language Version
_No response_
### Other information
_No response_
Contributor guide
Research direction
Start from the CDK CLI flow that handles DockerImageAsset builds and the image-publishing role; the issue does not name specific source files or tests. Reproduce the failure with the linked example and the private ECR Dockerfile, then verify that existing Docker credentials are used during the build while publishing-role credentials are used only for the asset push.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, docker, typescript
- Domain
- cloud, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100