File resolver indexes all files regardless of cataloger relevance — high baseline memory overhead
- Dominant language
- Go
- Stars
- 9.6k
- Forks
- 962
- Avg merge
- 23h 27m
- Merged PRs (30d)
- 48
Description
## Problem
The directory file resolver (`fileresolver.Directory`) walks the entire filesystem at initialization and builds a full in-memory index for every file — regardless of whether any enabled cataloger will ever query it. This includes a `FileNode` in the tree graph, an `IndexEntry` with metadata, MIME type detection (reading file content headers), and entries in 4 secondary index maps (by type, MIME, extension, basename).
This costs approximately 500-600 bytes per file. For large container images with hundreds of thousands of files, the file index alone consumes hundreds of megabytes before any cataloger runs.
## Profiling Data
We profiled syft v1.32.0 using `pprof` and `runtime.ReadMemStats` against extracted container image filesystems (simulating the directory source path):
| Image | Extracted Size | Files (approx) | File Index Baseline HeapInuse | Peak HeapInuse (serial) |
|-------|---------------|-----------------|-------------------------------|------------------------|
| alpine:latest | 8.7 MB | ~2K | ~30 MB | 271 MB |
| jenkins/jenkins:lts | 481 MB | ~30K | ~120 MB | 837 MB |
| gitlab/gitlab-ee:latest | 4.5 GB | ~1M | ~476 MB | 1,284 MB |
The gitlab-ee baseline of 476 MB is measured as HeapInuse before the first cataloger task runs — this is purely the file resolver index.
From pprof `alloc_space` on gitlab-ee, the top allocations during index construction:
- `mimetype.DetectReader` — 920 MB (reads file content headers for MIME type detection on every file)
- `stereoscope/pkg/filetree.(*FileTree).node` — 532 MB
- `stereoscope/pkg/filetree.(*FileTree).resolveAncestorLinks` — 422 MB
- `stereoscope/pkg/filetree.(*index).Add` — 247 MB
The MIME type detection is particularly costly — it reads content from every file to populate the `byMIMEType` index, yet only 2 catalogers use `FilesByMIMEType()` (`go-module-binary-cataloger` and `cargo-auditable-binary-cataloger`).
## Context
We use syft as a library in [Kubescape](https://github.com/kubescape/kubescape) node-agent and kubevuln components for in-cluster SBOM generation. These run as memory-constrained sidecars (1400 Mi for node-agent). The file index baseline alone can consume a third of the available memory before cataloging starts.
## Possible Directions
A few ideas — happy to discuss what makes sense for the project:
1. **Selective indexing:** If catalogers could declare their file requirements (globs, MIME types) upfront, the indexer could skip files that don't match any cataloger's patterns. The `generic.Cataloger` already captures these patterns via `WithParserByGlobs` / `WithParserByMimeTypes` — they'd just need to be queryable.
2. **Lazy MIME type detection:** Skip content-based MIME detection during indexing. The 2 catalogers that need it could detect executables via file permission bits or ELF magic bytes (4 bytes) instead of full MIME detection.
3. **Use `UnindexedDirectory` resolver:** Syft already has a lazy resolver that walks on-demand with near-zero memory. Its limitation is `FilesByMIMEType()` panics — if that were solved (e.g., via approach 2), it could be an option for memory-constrained environments.
We're happy to contribute to any of these approaches.
Contributor guide
Research direction
Start with the fileresolver.Directory and UnindexedDirectory resolvers, then trace generic.Cataloger options such as WithParserByGlobs and WithParserByMimeTypes and the FilesByMIMEType query. Compare their indexing and MIME-detection behavior against the profiling data. Done means reducing baseline memory without breaking catalogers that depend on file or MIME queries.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- performance, tooling
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100