Template builds are unbounded: `TemplateCreate` spawns one goroutine (and one FC VM + rootfs assembly) per request with no concurrency limit

Summary

(*ServerStore).TemplateCreate launches a detached build goroutine for every incoming gRPC request with no cap on how many run at once. Each goroutine drives a full template build: extracting an ext4 rootfs and booting a provisioning Firecracker VM, so N simultaneous TemplateCreate calls put N rootfs assemblies and N micro-VMs on a single template-manager host at the same time. Under a burst of builds this oversubscribes host disk, I/O, and RAM, with no backpressure anywhere in the path.

There's no upper bound and no queue: the only limiter is the rate at which clients call the API.

Where

packages/orchestrator/pkg/template/server/create_template.go

func (s *ServerStore) TemplateCreate(ctx context.Context, templateRequest *templatemanager.TemplateCreateRequest) (*emptypb.Empty, error) {
    // ... build BuildInfo, cache entry, logger ...

    s.wg.Add(1)
    s.activeBuilds.Add(1)
    go func(ctx context.Context) {           // <-- one goroutine per request, unbounded
        defer s.wg.Done()
        defer s.activeBuilds.Add(-1)
        // ...
        res, err := s.builder.Build(ctx, metadata, template, core)   // extracts ext4 + boots a provisioning FC VM
        // ...
    }(context.WithoutCancel(ctx))

    return nil, nil
}

server.New (main.go) wires no limiter into the ServerStore. Nothing bounds the number of concurrent builds.

Impact

When many builds are triggered close together (CI fan-out, a batch of templates, or several teams building at once), a single template-manager host runs all of them concurrently:

• N concurrent ext4 rootfs assemblies competing for disk space and I/O bandwidth
  in the build/cache directories.
• N concurrent provisioning Firecracker VMs competing for host RAM and CPU.
• No queue, so the host accepts work it can't safely run instead of deferring it.

Best case this is severe I/O/memory contention that slows every in-flight build and can trip build timeouts. Worst case the host runs out of disk mid-build and builds fail or produce corrupt rootfs images.

Production evidence (downstream self-host)

We hit this on a self-hosted deployment at build concurrency ~30 on one builder: the host exhausted local disk and a post-provision filesystem step raced, leaving corrupt rootfs images (e2fsck reporting block bitmap differences ... filesystem still has errors, exit 4) and failing the dependent provision scripts. A modest concurrency cap eliminated it entirely.

Steps to reproduce

1. Stand up a single template-manager / orchestrator builder host on EC2 c5.metal.
2. Fire a batch of TemplateCreate requests close together (e.g. 20–30 non-trivial templates) so they overlap.
3. Observe all builds start immediately and run concurrently: activeBuilds climbs to the batch size with no queueing. Watch build/cache disk usage and host memory spike, build wall-times degrade, and (on a sufficiently small host) builds fail.

Proposed fix

Bound concurrent builds with a weighted semaphore acquired inside the detached build goroutine, not in the TemplateCreate handler. The handler must stay non-blocking so the API's synchronous gRPC trigger returns immediately. Excess builds then queue (they already report as Building, which the API's status poll tolerates) instead of over-committing the host.

Sketch:

// in ServerStore (main.go), constructed in server.New:
buildLimiter *semaphore.Weighted   // golang.org/x/sync/semaphore

// inside the build goroutine in TemplateCreate, before s.builder.Build:
if err := s.buildLimiter.Acquire(ctx, 1); err != nil {
    // ctx cancelled while queued — fail the build cleanly via the cache entry
    buildInfo.SetFail(builderrors.UnwrapUserError(err))
    return
}
defer s.buildLimiter.Release(1)

Make the limit configurable. The idiomatic choice in this repo is a LaunchDarkly IntFlag in packages/shared/pkg/featureflags/flags.go, because that file already hosts a dense family of MaxConcurrent* / concurrency-cap flags and a template-build cap is the conspicuous gap in that family (see "Precedent" below):

// packages/shared/pkg/featureflags/flags.go, alongside BuildReservedDiskSpaceMB:
MaxConcurrentTemplateBuilds = NewIntFlag("max-concurrent-template-builds", 4)

A small default (≈4) bounds a single builder safely while leaving horizontal multi-builder scaling as a separate concern, and the flag stays runtime-tunable per environment without a redeploy.

Relationship to PR #3065 (build-reserved-disk-space) - complementary with this proposal

PR #3065 (feat(orchestrator): make build-reserved-disk-space-mb default 256MB) promotes BuildReservedDiskSpaceMB from 0 to 256 - i.e. it bounds how much disk a single build reserves on the guest root fs. That is orthogonal to this issue: it constrains per-build footprint, not the number of builds running at once. A builder can still be overwhelmed by N simultaneous builds even with the disk reservation in place. The two fixes compose: 1. reservation bounds each build's size, 2. the semaphore bounds the build count.

Notes

• Happy to open a PR with the semaphore + a feature flag + a test that asserts concurrency never exceeds the limit under a burst. Flagging direction here first per CONTRIBUTING.

Environment

• Repo: e2b-dev/infra
• Affected component: packages/orchestrator (template-manager build server)
• Observed against: main — defect re-verified present at commit 91e02e48a
  (2026-06-23): TemplateCreate still launches one unbounded go func per
  request (create_template.go:126), activeBuilds is still a debug-only
  counter read only for a log line (main.go), and no semaphore/limiter is
  wired into ServerStore. (Originally observed at 62aa3874f.)

Start with packages/orchestrator/pkg/template/server/create_template.go and trace ServerStore construction in main.go, then inspect the concurrency-cap flags in packages/shared/pkg/featureflags/flags.go. Verify how queued builds report status and add coverage for a burst of TemplateCreate requests. Done means concurrent builds never exceed the configured limit while the gRPC handler remains non-blocking.