GoogleContainerTools / GoogleContainerTools/skaffold

Build/Render/Run Fails with Kaniko + Immutable Tags

Open
#9,312 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
15.9k
Forks
1.7k
Avg merge
3d 9h
Merged PRs (30d)
10

Description

I have a project which uses kaniko to build an image remotely on a cluster. The kaniko build uses a cache repo and both repos have immutable tags configured. I use the inputDigest tagPolicy and set builds to be reproducible.

When I try to `skaffold run` this deployment and the tagPolicy would generate a tag which already exists, skaffold still tries to use kaniko to build the image. When kaniko pushes the image, because my repo has tag immutability configured it returns an error, kaniko fails the push, and the entire skaffold run is considered failed.

### Expected behavior

Builds with kaniko should check to see if a tag exists and skip the build if so.

### Actual behavior

When the image is built with kaniko, skaffold checks to see if the tag exists before starting the build, doesn't seem to find the existing tag, and it tries to build anyway.

### Information

- Skaffold version: v2.10.1
- Operating system: OS X 14
- Installed via: Homebrew
- Contents of skaffold.yaml:

```yaml
apiVersion: skaffold/v4beta9
kind: Config

build:
tagPolicy:
inputDigest: {}

artifacts:
- image: 987654321.dkr.ecr.us-west-1.amazonaws.com/my-repo
kaniko:
cache:
repo: 987654321.dkr.ecr.us-west-1.amazonaws.com/my-repo-cache
cacheCopyLayers: true
reproducible: true
skipUnusedStages: true
snapshotMode: redo
useNewRun: true
verbosity: info
whitelistVarRun: true

deploy:
helm:
releases:
- name: "my-app"
chartPath: deploy/my-app
valuesFiles:
- deploy/my-app/values.yaml
```

### Steps to reproduce the behavior

1. ```skaffold run```

```
$ skaffold run

Generating tags...
- 987654321.dkr.ecr.us-east-1.amazonaws.com/my-repo -> 987654321.dkr.ecr.us-west-1.amazonaws.com/my-repo:123456789
Checking cache...
- 987654321.dkr.ecr.us-west-1.amazonaws.com/my-repo: Not found. Building
Starting build...
Building [987654321.dkr.ecr.us-west-1.amazonaws.com/my-repo]...
Target platforms: [linux/amd64]
Flag --whitelist-var-run has been deprecated, Please use ignore-var-run instead.
time="2024-02-14T19:10:00Z" level=warning msg="Flag --snapshotMode is deprecated. Use: --snapshot-mode"
INFO[0000] Retrieving image manifest ...
[...build happens here, finds cached layers successfully, no problems...]
INFO[0331] Pushing image to 987654321.dkr.ecr.us-west-1.amazonaws.com/my-repo:123456789
error pushing image: failed to push to destination 987654321.dkr.ecr.us-west-1.amazonaws.com/my-repo:123456789: PUT https://987654321.dkr.ecr.us-west-1.amazonaws.com/v2/my-repo/manifests/123456789: TAG_INVALID: The image tag '123456789' already exists in the 'my-repo' repository and cannot be overwritten because the repository is immutable.
```

I can do a build with --dry-run and then check that the generated tag *does* exist before the build starts:

```
$ skaffold build --dry-run
Generating tags...
- 987654321.dkr.ecr.us-west-1.amazonaws.com/my-repo -> 987654321.dkr.ecr.us-west-1.amazonaws.com/my-repo:123456789
Skipping build phase since --dry-run=true

$ aws ecr list-images --repository-name ev-chargers --filter tagStatus=TAGGED --query 'imageIds[?imageTag==`'123456789'`]' --output json
[
{
"imageDigest": "sha256:a1s2d3f4g5h6j7k8l9",
"imageTag": "123456789"
}
]
```

#3849 *sounds* like a similar issue but I suspect it's just a matter of bad log messages--my cache repo is working correctly

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.