kubeflow / kubeflow/mcp-server

fix(trainer): inspect_controller reports RBAC and API errors as "no controller pod found"

Open
#245 1 comment 0 reactions 1 assignee Claimed by @yats0x7 View on GitHub
Dominant language
Python
Stars
44
Forks
54
Avg merge
2d 18h
Merged PRs (30d)
29

Description

### Description

`inspect_controller()` finds the trainer controller pod through `_find_controller_pod()` in `kubeflow_mcp/trainer/api/platform.py`. That helper lists pods for each candidate namespace and label selector, and wraps every call in `except Exception: continue`.

So when all of the lookups fail, the helper can't tell "there is no controller pod" apart from "I wasn't allowed to look" or "the API call failed". Each case comes back as no pod, and `inspect_controller` returns `RESOURCE_NOT_FOUND` with "No controller pod found in namespace 'kubeflow, kubeflow-system'".

RBAC is the realistic way to hit this. If the server's service account can't list pods in `kubeflow` or `kubeflow-system`, the admin is told the controller doesn't exist and goes looking for a missing deployment, when what's actually missing is a role binding. A 500 from the API server or a dropped connection gets reported the same way.

It also keeps real API failures away from the circuit breaker. `is_infrastructure_error()` only counts `KUBERNETES_ERROR`, `SDK_ERROR` and `TIMEOUT`, so a masked outage never registers.

One way to fix it is to remember the last exception while scanning. If no pod turns up, return `PERMISSION_DENIED` for a 403 or `KUBERNETES_ERROR` otherwise, with the underlying error in `details`. A pod found in any namespace should still win, so a 403 in one namespace doesn't hide a controller running in the other.

### Steps to Reproduce

```python
from unittest.mock import MagicMock, patch
from kubernetes.client.exceptions import ApiException
from kubeflow_mcp.trainer.api import platform

core = MagicMock()
core.list_namespaced_pod.side_effect = ApiException(status=403, reason="Forbidden")

with patch.object(platform.mcp_utils, "get_core_v1_api", return_value=core), \
patch.object(platform, "_get_controller_namespace", return_value=None):
print(platform.inspect_controller())
```

On a real cluster, run the server with a service account that has no permission to list pods in `kubeflow` and `kubeflow-system`, then call `inspect_controller()`.

### Expected Behavior

A 403 comes back as `PERMISSION_DENIED` and other API failures as `KUBERNETES_ERROR`, with the cause included. `RESOURCE_NOT_FOUND` only when the lookups worked and there really is no controller pod.

### Actual Behavior

Every failure is reported as a missing pod:

```
403 Forbidden -> RESOURCE_NOT_FOUND: No controller pod found in namespace 'kubeflow, kubeflow-system'
500 server error -> RESOURCE_NOT_FOUND: No controller pod found in namespace 'kubeflow, kubeflow-system'
connection refused -> RESOURCE_NOT_FOUND: No controller pod found in namespace 'kubeflow, kubeflow-system'
```

### MCP Server Version

0.1.1 (main)

### Python Version

3.12

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.