volcano-sh / volcano-sh/kthena

ModelBooster controller never starts on a workload only install

Open
#1,663 1 comment 0 reactions 1 assignee View on GitHub

@aeron-gh is already working on this.

Since Aug 20, 2026.

kind/bug
Dominant language
Go
Stars
460
Forks
201
Avg merge
4d 1h
Merged PRs (30d)
36

Description

/kind bug

Description

#1655 documents a control plane only install as a supported shape (--set networking.enabled=false), described as "a first-class scenario, not a degraded mode". On that install the networking CRDs are not present, and the ModelBooster controller does not currently handle that: it builds informers on ModelServer and ModelRoute unconditionally and then waits for those caches to sync, so its workers never start.

The process itself stays up and the ModelServing and autoscaler controllers keep working, so the deployment looks healthy. What is lost is that no ModelBooster is ever reconciled, and the log fills with list errors for the two missing resources.

Found by reading the source, not observed in a cluster. Repro steps are below for anyone who can run it.

Steps to reproduce the issue
  1. Install the workload subchart only, as the installation guide now describes:

    helm install kthena oci://ghcr.io/volcano-sh/charts/kthena \
      --namespace kthena-system --create-namespace \
      --set networking.enabled=false
    
  2. Watch the controller manager log. ModelServing controller started and Autoscaler controller started appear, start model controller does not, and reflector errors for *v1alpha1.ModelServer and *v1alpha1.ModelRoute repeat.

  3. Apply any ModelBooster. It is never reconciled and its status stays empty.

Evidence and production path

The default enables the controller. --controllers defaults to * (main.go#L82), and the chart leaves controllerManager.controllers empty (values.yaml#L28), so the flag is not passed and every controller including modelbooster runs.

NewModelBoosterController creates informers on both networking resources with no check that the CRDs exist:

https://github.com/volcano-sh/kthena/blob/172e0211878c456fdf354cfeb7e602a1bd1e072d/pkg/model-booster-controller/controller/model_booster_controller.go#L319-L320

Run starts them and then waits on their HasSynced:

https://github.com/volcano-sh/kthena/blob/172e0211878c456fdf354cfeb7e602a1bd1e072d/pkg/model-booster-controller/controller/model_booster_controller.go#L89-L90

https://github.com/volcano-sh/kthena/blob/172e0211878c456fdf354cfeb7e602a1bd1e072d/pkg/model-booster-controller/controller/model_booster_controller.go#L95-L101

With the CRDs absent, each reflector's initial List returns a 404 and retries, HasSynced stays false, so the wait does not return, start model controller is never logged and no worker starts. Since mc.Run is a goroutine (controller.go#L106), nothing crashes and the rest of the binary is unaffected.

Even past the sync, reconcile writes to both networking resources unconditionally:

https://github.com/volcano-sh/kthena/blob/172e0211878c456fdf354cfeb7e602a1bd1e072d/pkg/model-booster-controller/controller/model_booster_controller.go#L214-L221

which is expected, since ModelBooster is documented as cascading into both CRD groups.

The same binary already handles two optional CRDs this way, which is why this looked like an oversight rather than a design choice:

Describe the results you received and expected

Received: on a workload only install, the ModelBooster controller waits for caches that can never sync, so ModelBooster resources are silently not reconciled while the pod reports healthy.

Expected: the controller manager notices the networking CRDs are absent and skips the ModelBooster controller with a clear log line, the way it already does for LeaderWorkerSet, so that a control plane only install behaves as the installation guide describes.

If this looks right, I would be glad to send a PR using the existing ResourceExists helper (lws_controller.go#L380) to gate the ModelBooster controller the same way. Happy to take another approach if you prefer the controller to stay enabled and wait for the CRDs to appear later.

What version of Kthena are you using?

main at 172e021

Any other relevant information

Reported from source review on 172e021, the merge commit of #1655. Not reproduced on a cluster.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.