orchestrator: nbd read flood when build object missing from store (no negative caching + missing build_id in error log)

Summary

When the NBD overlay backend tries to read a block belonging to a build that no longer exists in the object store, two compounding problems produce a continuous high-frequency error flood in orchestrator_template-manager:

ERROR  nbd backend read failed  {"error": "error reading from device: failed to get build: ... failed to get object size: object does not exist", "nbd_op": "read", "nbd_provider": "*block.Overlay", "nbd_offset": 354713600, "nbd_length": 4096}

Observed rate: ~150+ errors/second from two trace IDs hitting the same missing offsets repeatedly.

Root Causes

1. No negative caching for missing builds (build/cache.go)

DiffStore.GetOrCreate uses singleflight to deduplicate concurrent calls but does not cache failed results. When createDiff returns storage.ErrObjectNotExist, the error is discarded and the next kernel retry calls createDiff again, resulting in a storage lookup (GCS/S3) on every NBD read retry.

// cache.go – GetOrCreate today
v, err, _ := s.initGroup.Do(string(key), func() (any, error) {
    diff, err := create(ctx)
    if err != nil {
        return nil, err  // error thrown away; next call re-runs create()
    }
    s.cache.Set(key, diff, ttlcache.DefaultTTL)
    return diff, nil
})

The Linux NBD kernel driver retries EIO-terminated block reads automatically, so once an object is missing the loop is: kernel retry → ReadAt → createDiff → storage 404 → EIO → kernel retry → …

2. Missing build ID in the error log (nbd/dispatch.go)

The nbd backend read failed log only records the NBD wire fields (nbd_offset, nbd_length, nbd_handle). The build UUID that caused the failure is buried in the error message string, making it impossible to quickly identify which build is missing without parsing free-text.

// dispatch.go – current log call
logger.L().Error(ctx, "nbd backend read failed",
    zap.Error(readErr),          // build ID only here, inside the error string
    zap.String("nbd_provider", d.provName),
    zap.Uint64("nbd_offset", from),
    // no zap.String("build_id", ...) field
)

Impact

• High cardinality error volume masks real issues in log aggregation
• Repeated storage (GCS/S3) 404 requests under kernel retry pressure
• On-call engineers cannot identify the affected build ID without grep-parsing the error string

Proposed Fix

1. Negative-cache ErrObjectNotExist in GetOrCreate

Introduce a short-lived negative cache (e.g., 60 s TTL via a sync.Map with expiry) so that a permanently-missing build returns immediately on subsequent reads without hitting storage.

2. Expose build ID as a structured log field in dispatch.go

Surface the build UUID (extractable from the wrapped error chain via errors.As) as a dedicated zap.String("build_id", ...) field so dashboards and alerting rules can group errors by build rather than by offset.

Affected Files

• packages/orchestrator/pkg/sandbox/build/cache.go – GetOrCreate
• packages/orchestrator/pkg/sandbox/nbd/dispatch.go – cmdRead error logger

Start with GetOrCreate in packages/orchestrator/pkg/sandbox/build/cache.go and cmdRead in packages/orchestrator/pkg/sandbox/nbd/dispatch.go. Trace the existing error types and cache behavior before checking the relevant Go package tests. Done means missing builds avoid repeated storage lookups for the intended TTL and read failures include a structured build_id field.