volcano-sh / volcano-sh/kthena
ModelBooster controller never starts on a workload only install
@aeron-gh is already working on this.
Since Aug 20, 2026.
- Dominant language
- Go
- Stars
- 460
- Forks
- 201
- Avg merge
- 4d 1h
- Merged PRs (30d)
- 36
Description
/kind bug
Description
#1655 documents a control plane only install as a supported shape (--set networking.enabled=false), described as "a first-class scenario, not a degraded mode". On that install the networking CRDs are not present, and the ModelBooster controller does not currently handle that: it builds informers on ModelServer and ModelRoute unconditionally and then waits for those caches to sync, so its workers never start.
The process itself stays up and the ModelServing and autoscaler controllers keep working, so the deployment looks healthy. What is lost is that no ModelBooster is ever reconciled, and the log fills with list errors for the two missing resources.
Found by reading the source, not observed in a cluster. Repro steps are below for anyone who can run it.
Steps to reproduce the issue
-
Install the workload subchart only, as the installation guide now describes:
helm install kthena oci://ghcr.io/volcano-sh/charts/kthena \ --namespace kthena-system --create-namespace \ --set networking.enabled=false -
Watch the controller manager log.
ModelServing controller startedandAutoscaler controller startedappear,start model controllerdoes not, and reflector errors for*v1alpha1.ModelServerand*v1alpha1.ModelRouterepeat. -
Apply any
ModelBooster. It is never reconciled and its status stays empty.
Evidence and production path
The default enables the controller. --controllers defaults to * (main.go#L82), and the chart leaves controllerManager.controllers empty (values.yaml#L28), so the flag is not passed and every controller including modelbooster runs.
NewModelBoosterController creates informers on both networking resources with no check that the CRDs exist:
Run starts them and then waits on their HasSynced:
With the CRDs absent, each reflector's initial List returns a 404 and retries, HasSynced stays false, so the wait does not return, start model controller is never logged and no worker starts. Since mc.Run is a goroutine (controller.go#L106), nothing crashes and the rest of the binary is unaffected.
Even past the sync, reconcile writes to both networking resources unconditionally:
which is expected, since ModelBooster is documented as cascading into both CRD groups.
The same binary already handles two optional CRDs this way, which is why this looked like an oversight rather than a design choice:
- LeaderWorkerSet: a discovery check disables the controller and logs it (lws_controller.go#L52-L58, controller.go#L96)
- PodGroup: the manager skips its informer when the CRD is missing (manager.go#L184-L186)
Describe the results you received and expected
Received: on a workload only install, the ModelBooster controller waits for caches that can never sync, so ModelBooster resources are silently not reconciled while the pod reports healthy.
Expected: the controller manager notices the networking CRDs are absent and skips the ModelBooster controller with a clear log line, the way it already does for LeaderWorkerSet, so that a control plane only install behaves as the installation guide describes.
If this looks right, I would be glad to send a PR using the existing ResourceExists helper (lws_controller.go#L380) to gate the ModelBooster controller the same way. Happy to take another approach if you prefer the controller to stay enabled and wait for the CRDs to appear later.
What version of Kthena are you using?
main at 172e021
Any other relevant information
Reported from source review on 172e021, the merge commit of #1655. Not reproduced on a cluster.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.