moby / moby/buildkit

Switch to more performant compression implementations

Open
#6,841 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area/exporter area/reproducible-builds exp/expert kind/bench-candidate kind/enhancement status/needs-investigation
Dominant language
Go
Stars
10.3k
Forks
1.5k
Avg merge
1d 23h
Merged PRs (30d)
48

Description

We now have support for compatibility-version that allows us to more easily make changes that change the checksum of the result artifact without breaking reproducible builds. This allows as to look at more performant compression options to speed up image layer creation, even if they produce slightly different artifacts.

Benchmarks in https://tonistiigi.github.io/compression-bench/

Gzip: klauspost-pgzip clear winner for gzip compression
Zstd: no significant performance difference between implementations (at least in CI)
Decompress: pigz cli wins, but gains are minimal (we already use this if pigz is available)

So obvious suggestion would be to switch to klauspost-pgzip. Problem though is that we export layers in parallel and don't have much control over limits or overallocation in this process. kp-pgzip gets its speed increase by doing much more memory allocations and then processing these in parallel from memory. From bench you can also see that there isn't much difference between using pgzip sequentially or in parallel for multiple layers. This means that when we process multiple layers in parallel, we allocate much more memory with no real performance benefit and in big builds could possible eventually run out of memory.

I'm not sure how serious this problem is in practice. Maybe it would be possible to extend kp-pgzip to control memory allocation across multiple writers.


The bench also shows that all the implementations seem to be deterministic, that is important for our reproducible builds. kp-pgzip looks deterministic in any concurrency level until you use the same fixed blocksize.

@fiam @AkihiroSuda

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the compatibility-version documentation and the compression benchmarks linked in the issue, then locate BuildKit's current layer compression and parallel export entry points. Compare candidate implementations for throughput, memory allocation, concurrency, and deterministic output. Done means a justified implementation choice that improves performance without unacceptable memory use or reproducibility regressions.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
build-system, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.