fix(api): background goroutines (events, builds) not awaited at graceful shutdown — events may be dropped

Summary

packages/api/main.go starts several background goroutines during request handling — sandbox event publishing, build status polling, cluster node health checks — but none of them are tracked in a WaitGroup or similar construct. When the API receives SIGTERM, the HTTP and gRPC servers are drained gracefully, but these background goroutines continue running with a cancelled context and are terminated mid-flight when os.Exit is called.

The code explicitly documents this gap:

// TODO: wait for additional work to coalesce
//
// currently we only wait for the HTTP handlers to return, and
// then cancel the remaining context and run all of the
// cleanup functions. Background go routines at this point
// terminate. Would need to have a goroutine pool or worker
// coordinator running to manage and track that work.

(packages/api/main.go:618)

Root cause

Goroutines that are spawned from HTTP handlers with context.WithoutCancel(ctx) (to decouple their lifetime from the request) are intentionally long-lived — but there is no mechanism to Wait() for them before the process exits.

Examples:

• go s.sbxEventsService.Publish(context.WithoutCancel(ctx), ...) in sandboxes.go — sandbox lifecycle events could be dropped on shutdown
• Metrics collection and health monitoring goroutines in api/main.go
• Background template build polling

Impact

On graceful shutdown:

1. In-flight event publishes are terminated mid-write → sandbox lifecycle events are silently dropped.
2. Build status updates may not reach their final state before the process exits.
3. Any in-progress DB writes from background goroutines are cut off without flushing.

This matters most during rolling deployments where the API is replaced and the old instance receives SIGTERM while sandboxes are still being created or killed.

Proposed fix

Introduce a sync.WaitGroup (or use golang.org/x/sync/errgroup) to track background goroutines spawned with context.WithoutCancel. The shutdown path (after HTTP/gRPC drain) should call wg.Wait() with a bounded timeout before os.Exit.

A lightweight approach is to wrap the existing go s.sbxEventsService.Publish(...) calls in a helper that increments/decrements the WaitGroup, and add wg.Wait() in the shutdown sequence after the HTTP servers close.

Read packages/api/main.go around line 618 and trace the shutdown sequence after the HTTP and gRPC servers drain. Then inspect the background publishes in sandboxes.go and the other WithoutCancel goroutines mentioned in the issue. Done means shutdown tracks this work and waits with a bounded timeout before exit, without dropping in-flight events or updates.