feat: detect and resolve middleware manifest drift with digest pinning and gateway refresh

Problem Statement

The gateway builds its middleware registry once at startup and caches every service manifest for its whole lifetime. The sandbox supervisor, by contrast, re-fetches manifests via Describe at sandbox startup and on every registry rebuild. When an operator upgrades a middleware service without restarting the gateway, the two views drift, and each direction of drift has a bad failure mode:

• Added binding: the gateway rejects policies referencing it ("not registered") even though the live service supports it. The only fix is a gateway restart.
• Removed or renamed binding: the gateway may still admit policies referencing the stale id (its cached manifest contains it; only a well-behaved service's live ValidateConfig catches it). On the supervisor's next registry rebuild the fresh manifest lacks the binding, the chain entry becomes unresolved, and a fail_open guard is bypassed per request. This is only observable after unguarded traffic has already flowed, via per-request Medium DetectionFindings that are indistinguishable from a transient service error, so nothing attributes the bypass to manifest drift.
• Changed max_body_bytes or config semantics: the supervisor keeps using the manifest captured at its last rebuild; a service that shrank its limit errors on oversized bodies and degrades per on_error.

There is currently no install-time detection of any of this and no way to refresh the gateway's view short of a restart.

Proposed Design

Three layers, each independently useful, ordered by cost:

1. Supervisor-side binding cross-check at registry rebuild (no protocol change)

Inside the combined policy+registry reload, after connect_services returns the freshly described registry, call ensure_policy_bindings_registered against the new registry for the incoming policy. A snapshot whose policy references bindings the live manifests no longer expose is rejected like any other reload failure: the last-known-good runtime pair stays installed, the snapshot is retried each poll, and the failure is loud (Medium OCSF config-state event plus failed policy status reported to the gateway). This converts the silent fail_open bypass at install time into a blocking, attributable signal.

2. Manifest digest pinning in the delivered registration (small, backward-compatible protocol addition)

Extend SupervisorMiddlewareService (sandbox.proto) with a gateway-stamped manifest_digest and the manifest's service_version:

• The digest is computed over the canonicalized manifest: bindings sorted by id, hashing id, operation, phase, and max_body_bytes. service_version is carried for diagnostics only (it is a free-form string and must not be the comparison key).
• During a registry rebuild, the supervisor compares the live manifest digest from its fresh Describe against the delivered digest. On mismatch it fails the snapshot (keep last-known-good, retry) and emits a DetectionFinding naming the registration, both digests, and both service versions, so operators see "the middleware contract changed underneath the gateway" rather than a generic connect error.
• Backward compatibility: an empty digest skips the check, so old gateways keep working with new supervisors and vice versa.

This also buys fleet convergence for free: the digest rides in the delivered registration, which is hashed into config_revision, so whenever the gateway's view of a manifest changes, every sandbox observes a service-set change on its next poll and performs a combined rebuild with a fresh Describe.

3. Gateway periodic manifest refresh

Re-Describe registered services on an interval, retaining the last-known-good manifest when a Describe fails so a flapping service cannot invalidate admitted policies. Combined with layer 2, a middleware upgrade propagates automatically: gateway refresh notices the new manifest, the delivered digest changes, config_revision changes, and every affected sandbox rebuilds against the new contract or loudly rejects it where it breaks the policy. This also removes the gateway-restart requirement for added bindings. Note one intentional behavior change: after a refresh removes a binding that admitted policies still reference, the per-delivery ensure_policy_bindings_registered check starts failing config delivery for those sandboxes (fail-safe, but it blocks their settings updates until the policy or service is fixed, and it should raise a gateway-side alert).

Alternatives Considered

• Pin service_version instead of a manifest digest: rejected as the comparison key. It is a free-form diagnostics string, so it over-detects (patch release with identical bindings) and under-detects (bindings changed without a version bump). Kept in the finding message for humans.
• Supervisor heartbeat re-Describe to catch mid-life drift: rejected. The per-request fail-open DetectionFinding already provides mid-life visibility, and layer 3 handles convergence at the right layer without a second polling loop per sandbox.
• Rely on the existing per-request fail-open findings alone (status quo): they fire only after unguarded requests have flowed, scale with traffic instead of with the drift event, and cannot be attributed to drift.
• Document "restart the gateway after middleware upgrades" (status quo): works but is easy to forget, and the failure mode for forgetting is a silently bypassed guard.

Agent Investigation

• Gateway registry is built once at startup and never refreshed: crates/openshell-server/src/lib.rs (hard failure if a registered service is unreachable at startup). Manifests are cached in a OnceCell per service, populated eagerly by connect_services.
• Policy admission validates against the cached manifest and calls the live ValidateConfig: crates/openshell-server/src/middleware.rs, MiddlewareRegistry::validate_policy_configs.
• Every config delivery re-checks ensure_policy_bindings_registered and filters required_services, both against the startup manifest snapshot: crates/openshell-server/src/grpc/policy.rs (get_sandbox_config).
• config_revision already hashes the delivered supervisor_middleware_services, which is what makes layer 2's convergence property work with zero new plumbing: compute_config_revision in crates/openshell-server/src/grpc/policy.rs.
• The supervisor rebuilds its registry with fresh Describe calls only when the delivered service set changes or the installed registry is degraded; policy-only changes swap the OPA engine alone (see PR #2027).
• The delivered registration currently carries only name, grpc_endpoint, and max_body_bytes: proto/sandbox.proto (SupervisorMiddlewareService).
• Per-request fail-open bypass finding that provides today's only drift visibility: crates/openshell-supervisor-network/src/l7/middleware.rs (middleware_events, "Middleware failed and was bypassed (fail_open)").

Start with the manifest and registry flow in crates/openshell-server/src/lib.rs and crates/openshell-server/src/middleware.rs, then trace delivery through crates/openshell-server/src/grpc/policy.rs and SupervisorMiddlewareService in proto/sandbox.proto. Review the supervisor rebuild path and the existing fail-open handling in crates/openshell-supervisor-network/src/l7/middleware.rs. Done means the proposed digest cross-check, safe refresh behavior, and gateway-to-sandbox convergence are implemented with drift reported and last-known-good state retained.