GoogleContainerTools / GoogleContainerTools/skaffold
`skaffold dev` and `skaffold run` are slow due to `Checking cache...` step
- Dominant language
- Go
- Stars
- 15.9k
- Forks
- 1.7k
- Avg merge
- 3d 9h
- Merged PRs (30d)
- 10
Description
### Given
We have the quite big frontend project with following structure:
- Frontend/Src - approx. 1,000 files.
- Frontend/Libs - approx. 20,000 files.
- Frontend/node_modules - approx. 80,000 files.
The `dockerfile` contains:
```
COPY ./Frontend /some/src
```
And a `dockerignore` file with 30 records:
```
# Exclude everything in the very beginning.
*
# Add only what is needed in this image.
!./Frontend
# Ignore these unneeded files if we occasionally added them above.
**/__pycache__
**/.cache
**/.coverage
**/.dockerignore
**/.DS_Store
**/.egg-info
**/.eggs
**/.git
**/.gitignore
**/.gitlab*
**/.mypy_cache
**/.pytest_cache
**/.vagrant
**/.vscode
**/*.egg-info
**/*.log
**/*.pyc
**/*.pyd
**/*.pyo
**/*.retry
**/*.sqlite3
**/*dockerfile*
**/dist
**/docker-compose*
**/node_modules
**/Vagrantfile
**/test
```
### Expected behavior
The Skaffold is able to run fast with this project structure. By fast I assume that Skaffold takes less than 10 seconds to process the given data.
### Actual behavior
The Skaffold is slow:
```
Cache check completed in 3 minutes ...
```
Both `skaffold run` and `skaffold dev` are suffering in performance badly: `skaffold dev` takes minutes to detect and sync file changes.
The docker itself is able to process the given number of files in seconds (with the docker build cache), while Skaffold is suffering for minutes.
### Information
- Skaffold version: v2.7.1 and v2.9.0 (Latest one, the v2.10.0 looks affected too)
- Operating system: Linux, Ubuntu 20.04.6
- Installed via: GitHub Releases
- Contents of skaffold.yaml: Not provided
Also:
- See the patch file contents below.
- Profiling for the `WalkWorkspace` function shows:
```
$ go tool pprof -text cpu-original.prof
Duration: 175.16s, Total samples = 178.60s (101.96%)
Showing nodes accounting for 168.66s, 94.43% of 178.60s total
Dropped 305 nodes (cum <= 0.89s)
flat flat% sum% cum cum%
62.32s 34.89% 34.89% 143.39s 80.29% regexp.(*Regexp).tryBacktrack
37.07s 20.76% 55.65% 37.07s 20.76% regexp.(*bitState).shouldVisit
18.66s 10.45% 66.10% 18.98s 10.63% regexp.(*bitState).push
15.74s 8.81% 74.91% 15.74s 8.81% regexp.(*inputString).step
7.35s 4.12% 79.03% 7.35s 4.12% regexp/syntax.(*Inst).MatchRunePos
```
It happens inside of the "github.com/moby/patternmatcher" while appling dockerignore rules to the given files.
### Steps to reproduce the behavior
1. Create a lot of rules in the dockerignore file (see the one provided)
2. Generate a lot of files, ignored by the dockerignore in the project folder.
3. Run the `skaffold run` command.
4. Measure the speed of the "Cache check" step.
5. Then apply the patch:
```
diff --git a/pkg/skaffold/docker/dependencies.go b/pkg/skaffold/docker/dependencies.go
index 05645da61..8560bf33e 100644
--- a/pkg/skaffold/docker/dependencies.go
+++ b/pkg/skaffold/docker/dependencies.go
@@ -24,8 +24,6 @@ import (
"runtime"
"sort"
- "github.com/moby/buildkit/frontend/dockerfile/dockerignore"
-
"github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/constants"
"github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/util"
"github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/walk"
@@ -215,52 +213,18 @@ func getDependenciesByDockerCopyFromTo(ctx context.Context, workspace string, do
// readDockerignore reads patterns to ignore
func readDockerignore(workspace string, absDockerfilePath string) ([]string, error) {
- var excludes []string
- dockerignorePaths := []string{
- absDockerfilePath + ".dockerignore",
- filepath.Join(workspace, ".dockerignore"),
- }
- for _, dockerignorePath := range dockerignorePaths {
- if _, err := os.Stat(dockerignorePath); !os.IsNotExist(err) {
- r, err := os.Open(dockerignorePath)
- if err != nil {
- return nil, err
- }
- defer r.Close()
-
- excludes, err = dockerignore.ReadAll(r)
- if err != nil {
- return nil, err
- }
- return excludes, nil
- }
- }
return nil, nil
}
// WalkWorkspace walks the given host directories and records all files found.
// Note: if you change this function, you might also want to modify walkWorkspaceWithDestinations.
func WalkWorkspace(workspace string, excludes, deps []string) (map[string]bool, error) {
- dockerIgnored, err := NewDockerIgnorePredicate(workspace, excludes)
- if err != nil {
- return nil, err
- }
-
// Walk the workspace
files := make(map[string]bool)
for _, dep := range deps {
absFrom := filepath.Join(workspace, dep)
-
keepFile := func(path string, info walk.Dirent) (bool, error) {
- if info.IsDir() && path == absFrom {
- return true, nil
- }
-
- ignored, err := dockerIgnored(path, info)
- if err != nil {
- return false, err
- }
- return !ignored, nil
+ return true, nil
}
if err := walk.From(absFrom).Unsorted().When(keepFile).Do(func(path string, info walk.Dirent) error {
```
6. Run the `skaffold run` command.
7. Measure the speed of the "Cache check" step again.
Contributor guide
Assessment
This issue has not been assessed yet.