moby / moby/buildkit

Add a per-build flag to limit concurrency / parallelism

Open
#6,038 2 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

status/triage
Dominant language
Go
Stars
10.3k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
50

Description

Description

Currently the only way to limit parallel builds is via config, added in https://github.com/moby/buildkit/pull/2049. It would be useful to be able to tune this per-build, for cases where parallel RUN commands are problematic, but only in that specific build and not system wide.

For example, I have a system where normally the default behavior is fine. However, if one stage is doing a very large download, it might choke downloads in other stages due to network congestion. This is a problem unique to the multi-stage Dockerfile that is trying to download a large model in one stage, and packages in another stage. I do not wish to retune my entire Docker daemon for this issue, so instead I'm simply disabling buildkit with DOCKER_BUILDKIT=0 docker build

This, or something very similar, has been hinted at but as far as I can tell not formally requested:

The current solution from https://github.com/moby/buildkit/pull/2049 does not meet this need, as that implementation is daemon-wide. The follow-up proposal in https://github.com/moby/buildkit/issues/2108 does not meet this need as there are times when the user will know better than the automated system, for example when there is a bottleneck on the internet link not easily detected. Please add a flag to docker build to allow tuning this per-build. Thank you.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the docker build flag/option path and the existing daemon-wide concurrency configuration described in the issue. Trace how a per-build limit would reach BuildKit, then verify that parallel RUN commands are limited for one build without changing daemon-wide behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
build-system, cli
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.