cloud-ark / cloud-ark/kubeplus

Support cross-namespace service dependencies for KubePlus Kind instances

Open
#1,481 1 comment 0 reactions 1 assignee Assigned to @prathameshk03 View on GitHub
Dominant language
Go
Stars
756
Forks
95
Avg merge
1d 14h
Merged PRs (30d)
7

Description

### Summary

KubePlus Kinds are registered from arbitrary Helm charts, so KubePlus has no built-in
understanding of a chart's `values.yaml` schema. When an instantiated Kind needs to reach
a shared service running in a different namespace, KubePlus currently has no mechanism to
provision the networking (and RBAC, where relevant) required for that cross-namespace call —
this has to be wired by hand today. We provide kubectl plugins for this. But it will be better if this can be done as part of the Operator.

This issue proposes a KubePlus-owned annotation, `kubeplus.io/cross-ns-deps`, that a Kind
instance can carry to declare its cross-namespace dependencies. KubePlus reconciles the
annotation into the NetworkPolicy objects (and namespace labels) needed to allow the traffic,
independent of what the underlying chart actually does with those values.

### Motivating example

- A `MCPServer` Kind is registered from a Helm chart and instantiated once, as
`shared-k8sgpt-mcp` in namespace `platform-mcp`.
- An `Agent` Kind (wrapping a kagent `Agent` + `ModelConfig`) is registered from a second
Helm chart, and instantiated **twice**, into two separate, isolated namespaces:
- `team-a-agent-instance` → namespace `team-a`
- `team-b-agent-instance` → namespace `team-b`
- Both `team-a` and `team-b` are otherwise network-isolated (default-deny), and neither
should be able to reach the other. Both should be able to reach `shared-k8sgpt-mcp` in
`platform-mcp`, and nothing else in `platform-mcp` beyond that one service.
- The Agent chart's `values.yaml` already has whatever fields it needs (e.g.
`mcpServer.name` / `mcpServer.namespace`) to template the kagent `Agent` CR's
namespace-qualified `RemoteMCPServer` reference. KubePlus is not involved in that part —
it only needs to open the network path.

### Annotation name and shape

Annotation key: `kubeplus.io/cross-ns-deps`

Placed on the **KubePlus Kind instance CR** (not on any resource rendered by the chart).
Value is a JSON array, allowing more than one dependency per instance from day one:

```yaml
apiVersion: cloudark.io/v1
kind: Agent
metadata:
name: team-a-agent-instance
namespace: team-a
annotations:
kubeplus.io/cross-ns-deps: |
[
{
"service": "shared-k8sgpt-mcp",
"namespace": "platform-mcp",
"port": 8089
}
]
spec:
mcpServer:
name: shared-k8sgpt-mcp
namespace: platform-mcp
modelConfigRef: default-model-config
systemMessage: "..."
```

Field definitions:

| Field | Required | Meaning |
|-------------|----------|--------------------------------------------------------------------------|
| `service` | yes | Name of the target Kubernetes Service (or Kind instance) being depended on |
| `namespace` | yes | Namespace the target lives in |
| `port` | no | If present, scope the NetworkPolicy egress/ingress rule to this port; if absent, allow all ports to the target namespace |

Note: `service`/`namespace` here duplicate whatever the chart's own `values.yaml` uses
(e.g. `spec.mcpServer.name`/`spec.mcpServer.namespace` above) — this is intentional.
KubePlus's dependency wiring stays chart-agnostic; the chart author is responsible for
keeping the two in sync, since only they know how their chart's `values.yaml` maps to the
running Service.

### How KubePlus should handle the annotation

**On Kind instance create/update:**

1. Watch for `kubeplus.io/cross-ns-deps` on any KubePlus Kind instance, across all
registered Kinds (this is Kind-agnostic — it's not specific to the `Agent` or
`MCPServer` Kind).
2. Parse the JSON array. For each entry:
- **Label the consumer namespace** (the instance's own namespace) with
`kubeplus.io/consumes..: "true"`
(or a hashed/short form of this if label value/key length limits are a concern —
labels are capped at 63 chars).
- **Create or update a deterministically-named egress `NetworkPolicy`** in the
consumer's namespace: `kubeplus-egress--`.
- `podSelector`: matches only the pods belonging to this instance (from the chart's
standard instance/app labels), **not** the whole namespace — even though today
each Agent instance has its own namespace, scoping to the instance's pods keeps
the policy correct if a namespace ever hosts more than one instance later.
- `egress`: allow to `namespaceSelector` matching `kubernetes.io/metadata.name:
`, restricted to `port` if given.
- **Ensure the target namespace carries a matching label** so its own ingress policy
can select on it, e.g. label the target namespace itself with
`kubeplus.io/shared-service: ` (idempotent — set once, reused by
every consumer).
- **Create or update one shared, deterministically-named ingress `NetworkPolicy` in
the target namespace**: `kubeplus-ingress-`.
- `podSelector`: matches the target service's pods.
- `ingress`: allow `from` any namespace carrying the corresponding
`kubeplus.io/consumes..: "true"` label (via
`namespaceSelector`), restricted to `port` if given.
- This object is shared across all consumers of the same target — it is **not**
rewritten per consumer; only the set of namespaces it selects grows as more
consumer-namespace labels appear. This avoids read-modify-write races between
concurrently reconciled Agent instances.
3. Requeue/reconcile on any change to the annotation value (new dependency added/removed).

**On Kind instance delete:**

1. Remove the consumer-namespace label(s)
(`kubeplus.io/consumes..`) that were set for this
instance's dependencies. If other instances in the same namespace still declare the
same dependency, leave the label in place (reference-count by checking for other
instances with the same dependency before removing).
2. Delete the per-instance egress `NetworkPolicy`
(`kubeplus-egress--`) — this one is always safe to
delete unconditionally since it's scoped to the deleted instance's own pods.
3. Do **not** delete the shared ingress `NetworkPolicy` in the target namespace — it
should persist as long as any consumer remains selected by it. If the label removal
in step 1 empties the set of matching namespaces, the policy remains present but
inert (matches nothing), which is fine; garbage-collecting the ingress policy object
itself is optional cleanup, not correctness-critical.

### RBAC note (separate from NetworkPolicy)

If the Kind's controller (or the underlying kagent controller) needs API-level read
access to the target object across namespaces (e.g. `get`/`watch` on the
`RemoteMCPServer` CR in `platform-mcp`), that's a second, smaller concern: a `RoleBinding`
in the target namespace granting the relevant ServiceAccount `get`/`list`/`watch` on that
specific object. Worth deciding whether this is in scope for `kubeplus.io/cross-ns-deps`
v1 or tracked as a follow-up — flagging it here so it isn't lost, but the NetworkPolicy
wiring above is the core ask.

### Acceptance criteria

- [ ] `kubeplus.io/cross-ns-deps` annotation is parsed on any KubePlus Kind instance
(Kind-agnostic).
- [ ] Creating an Agent instance with the annotation produces exactly one new egress
`NetworkPolicy` in the agent's namespace, scoped to that instance's pods.
- [ ] Creating a second Agent instance (different namespace, same target dependency)
results in the shared ingress `NetworkPolicy` in `platform-mcp` now selecting both
consumer namespaces, without modifying/duplicating the egress policies of either
agent.
- [ ] `team-a` and `team-b` cannot reach each other (neither declared a dependency on the
other).
- [ ] Deleting one Agent instance removes only its own egress policy and its own
namespace's consumption label; the other Agent instance's access to
`shared-k8sgpt-mcp` is unaffected.
- [ ] Deleting the last consumer leaves the shared ingress policy present but matching no
namespaces (no error state).

### Demo steps

1. **Set up cluster and register Kinds**
```bash
minikube start
kubectl create namespace platform-mcp
kubectl create namespace team-a
kubectl create namespace team-b
# Register the two Helm charts as KubePlus Kinds
kubectl apply -f mcpserver-kind.yaml
kubectl apply -f agent-kind.yaml
```

2. **Instantiate the shared MCP server**
```bash
kubectl apply -f - < -- curl -sS http://shared-k8sgpt-mcp.platform-mcp.svc.cluster.local:8089/mcp

# team-a cannot reach team-b
kubectl exec -n team-a -- curl -m 3 http://.team-b.svc.cluster.local # should time out
```

6. **Delete one Agent instance and show scoped cleanup**
```bash
kubectl delete agent team-a-agent-instance -n team-a
kubectl get networkpolicy -n team-a # egress policy gone
kubectl get networkpolicy -n platform-mcp -o yaml # ingress policy still present, now only selecting team-b
# team-b's access to shared-k8sgpt-mcp still works
kubectl exec -n team-b -- curl -sS http://shared-k8sgpt-mcp.platform-mcp.svc.cluster.local:8089/mcp
```

7. **Wrap-up talking point**: this is the same composition + relationship-tracking model
KubePlus already uses for SaaS instances, applied to agentic workloads — one annotation
contract, chart-agnostic, giving a platform team per-tenant agent isolation with shared,
governed access to common infrastructure (MCP servers, and later anything else a chart
author wants to declare a dependency on).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.