Agent-Hellboy / Agent-Hellboy/mcp-runtime

feat(operator): add platform readiness gate for MCPServer traffic

Open
#90 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
6
Forks
1
Avg merge
11h 33m
Merged PRs (30d)
13

Description

## Summary

Add platform-controlled readiness for MCPServer workloads so Kubernetes does not route traffic to a server until the MCP Runtime platform has verified it is actually ready, not just listening on a TCP port.

Today the operator-generated Deployment uses TCP liveness/readiness probes. That can mark a pod Ready while MCP-specific setup is still incomplete:

- MCP server process is listening, but not fully initialized.
- Tools/prompts/resources are not loaded yet.
- Gateway policy/auth configuration is not ready.
- Server has not registered or become visible to the platform/gateway.
- Observability/audit path is not connected.

This can send traffic too early and produce first-request failures.

## Candidate design

Use Kubernetes Pod readiness gates for operator-managed MCPServer pods.

Example generated PodSpec shape:

```yaml
spec:
readinessGates:
- conditionType: "mcpruntime.org/ready"
```

The MCP Runtime controller/operator would patch Pod status with a custom condition when platform-level checks pass:

```yaml
status:
conditions:
- type: "mcpruntime.org/ready"
status: "True"
reason: "MCPRuntimeReady"
message: "MCP server registered, gateway policy loaded, and platform checks passed"
```

Traffic should start only when both normal container readiness and the platform readiness gate are true.

## Platform readiness checks

Define what must be true before setting `mcpruntime.org/ready=True`. Suggested initial checks:

- MCP server container is Ready according to Kubernetes probe.
- Gateway sidecar is Ready when gateway is enabled.
- Gateway policy ConfigMap exists and matches the current MCPServer generation.
- MCP server responds to a lightweight MCP/platform health check, if available.
- Tool inventory is loaded or discoverable, if the server exposes a supported readiness endpoint.
- Required auth/session/policy config is present.
- Analytics/audit configuration is present when enabled.

Open design decision: avoid requiring optional external dependencies for readiness if their outage would make every tenant pod permanently unready. Some checks may belong in status warnings rather than the readiness gate.

## Operator behavior

- Add readiness gates to stable and canary MCPServer Deployments.
- Watch owned Pods or reconcile them through the MCPServer reconciliation loop.
- Patch Pod status conditions through the Kubernetes status subresource.
- Set `mcpruntime.org/ready=False` while platform checks are pending or failed.
- Set `mcpruntime.org/ready=True` when checks pass.
- Include reason/message for debugging.
- Clear/recompute the condition when MCPServer generation, gateway policy, image, or relevant config changes.
- Update MCPServer status to reflect platform-readiness failures separately from Deployment readiness.

## Safety considerations

- Readiness gates can deadlock rollout if the controller cannot patch Pod status. The implementation needs clear failure modes and operator logs/events.
- If the operator is down, new pods may remain unready. Document this and consider whether the feature should be configurable.
- Do not let tenant-controlled workloads set or spoof the readiness gate condition.
- Keep the condition type under the project domain, for example `mcpruntime.org/ready`.
- Consider a feature flag/operator env var for rollout:
- `MCP_ENABLE_PLATFORM_READINESS_GATE=true|false`

## API / CRD considerations

Possible MCPServer status additions:

```yaml
status:
platformReady: true
conditions:
- type: PlatformReady
status: "True"
reason: MCPRuntimeReady
```

Possible MCPServer spec option if per-server override is needed later:

```yaml
spec:
readiness:
platformGate: true
```

Default should probably be platform-controlled, not tenant-controlled, for hosted/multi-tenant environments.

## Tests

Add tests for:

- Generated Deployment includes `readinessGates` when enabled.
- Stable and canary Deployments both include the gate.
- Operator patches Pod condition to `False` while platform checks are pending.
- Operator patches Pod condition to `True` when platform checks pass.
- MCPServer status exposes platform readiness failure reasons.
- Rollout does not report fully ready until platform readiness is true.
- Feature can be disabled if a compatibility flag is added.

## Documentation

Update docs to explain:

- Kubernetes readiness probes vs MCP Runtime platform readiness.
- Why pods can listen before MCP tools/auth/gateway state is ready.
- How to debug `mcpruntime.org/ready=False`.
- Operator events/logs to inspect.
- Whether the readiness gate is enabled by default.

## Acceptance criteria

- MCPServer pods are not considered Ready until MCP Runtime platform checks pass.
- Traffic does not reach a pod that has only passed TCP/container readiness but has not passed platform readiness.
- `kubectl describe pod` shows a clear custom readiness gate condition and reason/message.
- MCPServer status reflects platform readiness state.
- Tests cover deployment rendering and condition patching behavior.
- Docs cover debugging and operational tradeoffs.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.