anchore / anchore/syft

High memory consumption when cataloging images with many files

Open
#2,159 5 comments 4 reactions 1 assignee Claimed by @kzantow View on GitHub
bug performance
Dominant language
Go
Stars
9.6k
Forks
954
Avg merge
1d 5h
Merged PRs (30d)
42

Description

**What happened**:
While running an anchore/syft container in k8s with a 1GB memory limit, I noticed that it was killed on OOM while calculating the SBOM for the following demo image:
`public.ecr.aws/ciscoeti/apa/bestbags-delivery@sha256:a560bdbb72f563b4be354f414aa812d442a9c1a0527d6687f95f67c8c57a65bc`

Created a simple go binary to profile syft package cataloging to understand what causes the high memory consumption:
https://gist.github.com/akpsgit/bc14660363f3217c9df8d1e8900076cf

Got the following memory peak towards the end of cataloging:
```
Alloc = 2 MiB TotalAlloc = 4 MiB Sys = 14 MiB NumGC = 1
Alloc = 2 MiB TotalAlloc = 4 MiB Sys = 14 MiB NumGC = 1
Alloc = 2 MiB TotalAlloc = 4 MiB Sys = 14 MiB NumGC = 1
Alloc = 29 MiB TotalAlloc = 118 MiB Sys = 40 MiB NumGC = 24
Alloc = 34 MiB TotalAlloc = 200 MiB Sys = 53 MiB NumGC = 29
Alloc = 39 MiB TotalAlloc = 294 MiB Sys = 78 MiB NumGC = 33
Alloc = 58 MiB TotalAlloc = 372 MiB Sys = 87 MiB NumGC = 35
Alloc = 71 MiB TotalAlloc = 559 MiB Sys = 131 MiB NumGC = 40
Alloc = 110 MiB TotalAlloc = 701 MiB Sys = 140 MiB NumGC = 42
Alloc = 125 MiB TotalAlloc = 893 MiB Sys = 165 MiB NumGC = 45
Alloc = 130 MiB TotalAlloc = 1120 MiB Sys = 227 MiB NumGC = 48
Alloc = 133 MiB TotalAlloc = 1335 MiB Sys = 256 MiB NumGC = 50
Alloc = 353 MiB TotalAlloc = 1717 MiB Sys = 383 MiB NumGC = 52
Alloc = 816 MiB TotalAlloc = 2474 MiB Sys = 986 MiB NumGC = 54
2023/09/20 16:57:04 More than 800 MB is allocated!!! dumping memory profile
Alloc = 782 MiB TotalAlloc = 2753 MiB Sys = 1084 MiB NumGC = 55
```
From lookin at the memory graph of ``` go tool pprof -http=:8080 ./20230920165704-mem.prof ```:
In use space:
![image](https://github.com/anchore/syft/assets/39153880/acc824bc-fb53-458e-a8ff-3240832426e1)

Allocated space:
![image](https://github.com/anchore/syft/assets/39153880/a37691ed-064d-4352-acce-76db09e8a760)

looks like the issue might be related to the file tree squash or the mimetype DetectReader().

From looking at the image content to see what special about it that can explain the high consumption, looks like it has ~40K files, which can explain the high usage in tree squash and mimetype DetectReader.

**What you expected to happen**:

**Steps to reproduce the issue**:
1. Build and run the attached go gist: https://gist.github.com/akpsgit/bc14660363f3217c9df8d1e8900076cf
2. Check for the following print "2023/09/20 16:57:04 More than 800 MB is allocated!!! dumping memory profile"
3. Check that a memory profile was created in the directory (e.g. 20230920165704-mem.prof)
4. Examine the profile: ``` go tool pprof -http=:8080 ./20230920165704-mem.prof ```

**Anything else we need to know?**:
Might be related:
https://github.com/gabriel-vasile/mimetype/issues/354

**Environment**:
- Output of `syft version`:
```
docker run --rm -it anchore/syft version
Application: syft
Version: 0.90.0
BuildDate: 2023-09-11T21:22:00Z
GitCommit: b82c0ffc3417bdc8c38f4633af95a668ec29fa35
GitDescription: v0.90.0
Platform: linux/amd64
GoVersion: go1.21.0
Compiler: gc
```
- OS (e.g: `cat /etc/os-release` or similar):
K8s kind, MacOS

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.