CLI flag to limit the number of concurrently building targets in buildx bake / docker compose build

Description

Description

There is currently no per-invocation, command-line way to limit build concurrency when building multiple targets with docker buildx bake (or docker compose build, which since Compose v5.0.0 delegates all builds to Bake). All targets are dispatched to the builder at once, and the only existing control (max-parallelism in the BuildKit worker configuration) must be baked into the builder at creation time.

I am requesting a client-side concurrency limit, e.g.:

docker buildx bake --max-parallelism 2 -f docker-compose.yml
docker compose build --max-parallelism 2

that caps how many targets (build requests) are in flight simultaneously, independent of which driver or daemon configuration is in use.

Background / motivating use case

CI pipeline (Jenkins on Kubernetes, ephemeral Docker-in-Docker pods) building multi-service applications from a Compose file. Each service is a multi-stage Dockerfile with resource-heavy compile steps (webpack/node, .NET, etc.).

Different applications sharing the same pipeline have very different resource profiles, so the desired parallelism is a per-project, per-invocation value supplied by the CI system — not a property of the builder or the daemon.

With the historical Compose builder this was trivial:

COMPOSE_PARALLEL_LIMIT=2 docker compose build

That capability has been progressively removed:

1. Compose ≥ v2.34 defaults to the Bake backend; COMPOSE_PARALLEL_LIMIT / --parallel no longer affect builds (docker/compose#13043, docker/desktop-feedback#584).
2. COMPOSE_BAKE=false was deprecated in Compose v2.39.0 (docker/compose#13065).
3. Compose v5.0.0 removed the internal builder entirely (docker/compose#13056), so the last escape hatch is gone.

There is now no supported equivalent of the old behavior anywhere in the CLI surface.

Current workaround and why it is insufficient

The documented approach is to set worker-level parallelism when creating a builder:

docker buildx create --name ci --driver docker-container \
  --buildkitd-flags '--oci-max-parallelism=2' --bootstrap --use

It has significant drawbacks:

• Not per-invocation. The value is frozen at builder-creation time. Changing it means destroying and recreating the builder (losing its local cache) or pre-provisioning one builder per concurrency tier and selecting with --builder.
• Not available on the default docker driver. The BuildKit instance embedded in dockerd exposes no max-parallelism in daemon.json and no dockerd flag. Users on the default driver have no option at all.
• Forces the docker-container driver in environments where its costs are highest. In ephemeral CI (DinD pods) this means creating a builder per job and paying the --load export/import penalty (minutes for large images) or restructuring pipelines around --push, purely to obtain a concurrency limit.

Proposal

1. --max-parallelism <N> on docker buildx bake (and build for completeness)

Client-side scheduling: Bake already computes the target dependency graph (contexts = { x = "target:y" }); instead of dispatching all ready targets at once, dispatch at most N concurrently. Because this is done in the CLI before solves are submitted:

• it works with every driver, including the default docker driver;
• it requires no daemon reconfiguration or builder lifecycle changes;
• it composes with (does not replace) worker-level max-parallelism, which remains the right tool for capping step concurrency inside a single large multi-stage build.

2. Bake-file attribute for per-group/per-target defaults

group "default" {
  targets = ["app-a", "app-b", "app-c"]
  max-parallelism = 2
}

with the CLI flag taking precedence, consistent with how other bake attributes are overridden via --set.

3. Compose passthrough

docker compose build --max-parallelism N (and/or reviving COMPOSE_PARALLEL_LIMIT mapped onto it), restoring pre-Bake parity now that Compose v5 has no internal builder.

Prior art / related issues

• docker/buildx#2581 — Add concurrency control to matrix builds (open; proposes --max-parallel and a per-target concurrency attribute)
• moby/buildkit#6038 — Add a per-build flag to limit concurrency/parallelism (open; notes the docker/for-win#2049 daemon-wide config "does not meet this need")
• moby/buildkit#5981 — Specify max parallelism for multi-stage build (open)
• docker/buildx#3006 — Limit concurrent build requests on a single BuildKit pod (open; worker max-parallelism does not cap accepted build requests)
• docker/compose#13043 — COMPOSE_PARALLEL_LIMIT no longer limiting concurrent builds
• docker/desktop-feedback#584 — All parallelism-limiting mechanisms ineffective once Bake is the builder (100 services building simultaneously)
• moby/buildkit#2049 — original daemon-wide max-parallelism config (shipped; explicitly daemon-scoped)

The recurring theme across these issues: the only shipped control is daemon-scoped and creation-time-static, while the demand is for an invocation-scoped, client-side limit.

Expected behavior

docker buildx bake --max-parallelism 1 -f compose.yml svc-a svc-b

builds svc-a and svc-b strictly sequentially (subject to dependency order), on any driver, without any builder or daemon reconfiguration. Default remains unlimited (current behavior), so the flag is fully backward compatible.

Environment where this was hit

• Docker Engine 28.3.x (DinD), Buildx ≥ v0.23, Compose v5.3.1
• Jenkins on Kubernetes, ephemeral DinD build pods, one builder per job

Start by tracing how buildx bake computes its target dependency graph and dispatches ready targets. Compare that scheduling path with the related concurrency-control issues named here, then verify that an invocation limit preserves dependency order, caps in-flight targets, and leaves the default unlimited.