serviceoffer-controller silently reverts per-agent Hermes config (provider, MCP servers, turn cap) on every reconcile/upgrade

Summary

The serviceoffer-controller silently reverts each Agent's hermes-config (its LLM inference provider, MCP-server wiring, and turn cap) on every reconcile and every controller restart — including every obol stack up that bumps the controller image. Operator configuration that isn't expressible in the Agent CRD is destroyed with no warning.

This is not theoretical: after a routine silvernuc3 upgrade it took a paying agent endpoint down with a 502 (agent reverted from its configured provider to a backend that had no healthy deployment; the stale API server then returned EOF → the x402 verifier surfaced it as 502).

Root cause

Three compounding facts (all at internal/serviceoffercontroller/):

1. renderHermesConfig(model, litellmKey) (agent_render.go:120) emits a fixed template. Only the model name is variable — provider: custom→litellm, max_turns: 30, disabled_toolsets: [memory, web] are hardcoded, and no mcp_servers block is ever emitted. So an operator's grok/xai-oauth provider, raised turn cap, or paid MCP tools exist only until the next render overwrites them.

2. The hermes-config ConfigMap is full-replaced on every reconcile (applyAgentObject, agent.go:449 — get-then-full-Update, despite a stale "server-side-applied" comment). There is no content-hash skip on this path (one exists for the skill catalog, catalog.go, but was never applied here).

3. The informer's initial List on every controller-pod start re-emits Add for every Agent → a full re-provision of every agent. Since the controller image is digest-pinned and bumped per release, every upgrade restarts the controller → re-provisions every agent → reverts operator config.

The AgentSpec also has no way to express provider / MCP servers / turn cap, so there is no supported path for an operator to make this config survive.

Reproduce

1. Configure an agent with a non-custom provider (e.g. xai-oauth) and/or an mcp_servers block by editing the hermes-config ConfigMap directly.
2. Restart serviceoffer-controller (or obol stack up).
3. Observe the ConfigMap reverts to the template (gemma/litellm, no MCP), and the agent breaks.

Proposed fix

Make the Agent CR the source of truth, and stop rewriting unchanged config:

• A — Additive CRD fields + render passthrough. Add optional modelProvider, mcpServers, maxTurns, disabledToolsets to AgentSpec (keep model string; no version bump/conversion) and render them. Unset → byte-identical to today.
• B — Hash-skip. Stamp obol.org/hermes-config-hash on the live ConfigMap and skip the apply when the freshly-rendered desired hash matches — survives controller restarts, so unchanged config is never rewritten. Mirrors the existing catalog.go pattern.
• C — ConfigDrift status condition. Surface out-of-band ConfigMap edits instead of silently clobbering them (the safety net for B, which intentionally stops self-healing drift).

PR implementing A→B→C follows and has been validated live on silvernuc3 (controller restart no longer changes the ConfigMap resourceVersion; grok+MCP config preserved; agent healthy).

Start in internal/serviceoffercontroller/agent_render.go at renderHermesConfig and internal/serviceoffercontroller/agent.go at applyAgentObject; compare the existing hash-skip pattern in catalog.go. Review the Agent CRD and controller tests before implementing the proposed fields, hash handling, and ConfigDrift condition. Done means reconciles and restarts preserve supported configuration and report out-of-band edits without silently overwriting them.