GoogleContainerTools / GoogleContainerTools/skaffold

skaffold dependency cache is incorrect after host before hooks run and are sent to docker API

Open
#8,791 1 comment 0 reactions 0 assignees View on GitHub
area/cache kind/bug
Dominant language
Go
Stars
15.9k
Forks
1.7k
Avg merge
3d 9h
Merged PRs (30d)
10

Description

If you use `useDockerCLI: false` you cannot have before hooks that move/delete files that your Dockerfile needs. We currently use a before hook to copy in generated protobuf code into a Go container and that's how we discovered this issue. I have made a dummy repo that replicates this issue very simply here: https://github.com/SophisticaSean/skaffold-docker-hook-bug

Simply toggling `useDockerCLI` off and on showcases the issue. With the `useDockerCLI` enabled there's no way for us to pass file dependency information to the docker CLI so this issue is not present. If you disable file dependency caching by changing [this line](https://github.com/GoogleContainerTools/skaffold/blob/main/pkg/skaffold/graph/dependencies.go#L91) to this:

```golang
// return getDependenciesFunc(ctx, a, r.cfg, r.artifactResolver)
return []string{}, nil
```
you then can use `useDockerCLI: true` and the build will succeed.

I believe that `skaffold dev` works around this by forcibly rebuilding the cache on filepath/state change [here.](https://github.com/GoogleContainerTools/skaffold/blob/main/pkg/skaffold/runner/listen.go#L92)

Since the hooks are embedded in the skaffold build image function and that happens _after_ tagging, and tagging is where this file dependency cache is built, before hooks cannot currently update/change the already cached docker file dependencies. I'm unsure of the next steps forward to fix this issue.

I'm happy to contribute a PR if y'all could tell me _how_ you want to fix this/general approach.

### Expected behavior
Should be able to have before hooks that move files around and have a docker API build succeed.
With `useDockerCLI` enabled:
![image](https://github.com/GoogleContainerTools/skaffold/assets/2059265/e82f092e-d728-4d07-880b-2a031a72d46a)

### Actual behavior
With `useDockerCLI` disabled (docker API build):
![image](https://github.com/GoogleContainerTools/skaffold/assets/2059265/edb17a65-44c4-47ba-b1c5-73db140ef42d)

### Information

- Skaffold version: v2.4.0
- Operating system: Ubuntu 24, macOS, ubiquitous.
- Installed via: skaffold.dev, and compiled from source (main/master).
- Contents of skaffold.yaml:

https://github.com/SophisticaSean/skaffold-docker-hook-bug/blob/main/skaffold.yaml
```yaml
apiVersion: skaffold/v4beta1
kind: Config
metadata:
name: skaffold-bug
build:
artifacts:
- image: example
context: example
hooks:
before:
- command: ["bash", "-c", "./example/skaffold_before_hook.sh"]
local:
concurrency: 0
tryImportMissing: false # THIS HAS TO BE FALSE OR CACHE BUSTING WONT WORK
useDockerCLI: false # THIS HAS TO BE TRUE OR THE BEFORE SCRIPT WONT WORK
```

### Steps to reproduce the behavior

1. git clone https://github.com/SophisticaSean/skaffold-docker-hook-bug
2. change `useDockerCLI` in skaffold.yaml to false/true
3. `/bin/rm -rf ./example/cool-files-copy/needed-file.txt; skaffold build --cache-artifacts=true -b example`
4. observe with `useDockerCLI` true, it works, with false it fails saying the file `needed-file.txt` doesn't exist in the docker container.

Logs are not relevant, they don't show anything out of the ordinary as caching doesn't get logged out.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.