kubeflow / kubeflow/mcp-server
fix(trainer): inspect_controller reports RBAC and API errors as "no controller pod found"
- Dominant language
- Python
- Stars
- 44
- Forks
- 54
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 29
Description
### Description
`inspect_controller()` finds the trainer controller pod through `_find_controller_pod()` in `kubeflow_mcp/trainer/api/platform.py`. That helper lists pods for each candidate namespace and label selector, and wraps every call in `except Exception: continue`.
So when all of the lookups fail, the helper can't tell "there is no controller pod" apart from "I wasn't allowed to look" or "the API call failed". Each case comes back as no pod, and `inspect_controller` returns `RESOURCE_NOT_FOUND` with "No controller pod found in namespace 'kubeflow, kubeflow-system'".
RBAC is the realistic way to hit this. If the server's service account can't list pods in `kubeflow` or `kubeflow-system`, the admin is told the controller doesn't exist and goes looking for a missing deployment, when what's actually missing is a role binding. A 500 from the API server or a dropped connection gets reported the same way.
It also keeps real API failures away from the circuit breaker. `is_infrastructure_error()` only counts `KUBERNETES_ERROR`, `SDK_ERROR` and `TIMEOUT`, so a masked outage never registers.
One way to fix it is to remember the last exception while scanning. If no pod turns up, return `PERMISSION_DENIED` for a 403 or `KUBERNETES_ERROR` otherwise, with the underlying error in `details`. A pod found in any namespace should still win, so a 403 in one namespace doesn't hide a controller running in the other.
### Steps to Reproduce
```python
from unittest.mock import MagicMock, patch
from kubernetes.client.exceptions import ApiException
from kubeflow_mcp.trainer.api import platform
core = MagicMock()
core.list_namespaced_pod.side_effect = ApiException(status=403, reason="Forbidden")
with patch.object(platform.mcp_utils, "get_core_v1_api", return_value=core), \
patch.object(platform, "_get_controller_namespace", return_value=None):
print(platform.inspect_controller())
```
On a real cluster, run the server with a service account that has no permission to list pods in `kubeflow` and `kubeflow-system`, then call `inspect_controller()`.
### Expected Behavior
A 403 comes back as `PERMISSION_DENIED` and other API failures as `KUBERNETES_ERROR`, with the cause included. `RESOURCE_NOT_FOUND` only when the lookups worked and there really is no controller pod.
### Actual Behavior
Every failure is reported as a missing pod:
```
403 Forbidden -> RESOURCE_NOT_FOUND: No controller pod found in namespace 'kubeflow, kubeflow-system'
500 server error -> RESOURCE_NOT_FOUND: No controller pod found in namespace 'kubeflow, kubeflow-system'
connection refused -> RESOURCE_NOT_FOUND: No controller pod found in namespace 'kubeflow, kubeflow-system'
```
### MCP Server Version
0.1.1 (main)
### Python Version
3.12
Contributor guide
Assessment
This issue has not been assessed yet.