chainguard-dev / chainguard-dev/apko

tarfs: Entry retains a full tar.Header per file; store a compact record

Open
#2,492 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
1.7k
Forks
228
Avg merge
1d 3h
Merged PRs (30d)
62

Description

tarfs.New (pkg/apk/expandapk/tarfs/tarfs.go:207) keeps an Entry per tar member holding a full tar.Header copy, including PAXRecords and Xattrs maps and the header's string fields, plus a fs.FileInfo. Callers that cache expanded packages keep these indexes alive for the process lifetime, so every file of every cached package carries the full header.

In a 7-day averaged heap profile of a long-running process, tarfs.New accounts for 0.9 GB in-use, of which archive/tar.parsePAX is 350 MB and readHeader another 260 MB. Those are PAX maps and header strings retained after parsing, not the index itself.

A compact entry (name, size, mode, uid/gid, mod time, typeflag, linkname, offset) with the PAX map dropped once the header is resolved would likely halve the row. FileInfo can be derived on demand from the compact record. Anything that needs a full tar.Header back can re-read it from the offset, which is what the reader path already does for content.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at tarfs.New in pkg/apk/expandapk/tarfs/tarfs.go:207 and trace how Entry, fs.FileInfo, and tar.Header are used by the reader path. Measure the retained header data and identify the existing offset-based header reread behavior. Done means entries retain only the needed compact fields, PAX data is released after resolution, and callers can still obtain file metadata and content.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
performance
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.