GoogleContainerTools / GoogleContainerTools/skaffold
skaffold dependency cache is incorrect after host before hooks run and are sent to docker API
- Dominant language
- Go
- Stars
- 15.9k
- Forks
- 1.7k
- Avg merge
- 3d 9h
- Merged PRs (30d)
- 10
Description
If you use `useDockerCLI: false` you cannot have before hooks that move/delete files that your Dockerfile needs. We currently use a before hook to copy in generated protobuf code into a Go container and that's how we discovered this issue. I have made a dummy repo that replicates this issue very simply here: https://github.com/SophisticaSean/skaffold-docker-hook-bug
Simply toggling `useDockerCLI` off and on showcases the issue. With the `useDockerCLI` enabled there's no way for us to pass file dependency information to the docker CLI so this issue is not present. If you disable file dependency caching by changing [this line](https://github.com/GoogleContainerTools/skaffold/blob/main/pkg/skaffold/graph/dependencies.go#L91) to this:
```golang
// return getDependenciesFunc(ctx, a, r.cfg, r.artifactResolver)
return []string{}, nil
```
you then can use `useDockerCLI: true` and the build will succeed.
I believe that `skaffold dev` works around this by forcibly rebuilding the cache on filepath/state change [here.](https://github.com/GoogleContainerTools/skaffold/blob/main/pkg/skaffold/runner/listen.go#L92)
Since the hooks are embedded in the skaffold build image function and that happens _after_ tagging, and tagging is where this file dependency cache is built, before hooks cannot currently update/change the already cached docker file dependencies. I'm unsure of the next steps forward to fix this issue.
I'm happy to contribute a PR if y'all could tell me _how_ you want to fix this/general approach.
### Expected behavior
Should be able to have before hooks that move files around and have a docker API build succeed.
With `useDockerCLI` enabled:

### Actual behavior
With `useDockerCLI` disabled (docker API build):

### Information
- Skaffold version: v2.4.0
- Operating system: Ubuntu 24, macOS, ubiquitous.
- Installed via: skaffold.dev, and compiled from source (main/master).
- Contents of skaffold.yaml:
https://github.com/SophisticaSean/skaffold-docker-hook-bug/blob/main/skaffold.yaml
```yaml
apiVersion: skaffold/v4beta1
kind: Config
metadata:
name: skaffold-bug
build:
artifacts:
- image: example
context: example
hooks:
before:
- command: ["bash", "-c", "./example/skaffold_before_hook.sh"]
local:
concurrency: 0
tryImportMissing: false # THIS HAS TO BE FALSE OR CACHE BUSTING WONT WORK
useDockerCLI: false # THIS HAS TO BE TRUE OR THE BEFORE SCRIPT WONT WORK
```
### Steps to reproduce the behavior
1. git clone https://github.com/SophisticaSean/skaffold-docker-hook-bug
2. change `useDockerCLI` in skaffold.yaml to false/true
3. `/bin/rm -rf ./example/cool-files-copy/needed-file.txt; skaffold build --cache-artifacts=true -b example`
4. observe with `useDockerCLI` true, it works, with false it fails saying the file `needed-file.txt` doesn't exist in the docker container.
Logs are not relevant, they don't show anything out of the ordinary as caching doesn't get logged out.
Contributor guide
Assessment
This issue has not been assessed yet.