kubeflow / kubeflow/trainer

Flux plugin panics on nil NumNodes dereference

Open
#3,667 2 comments 0 reactions 1 assignee Claimed by @AdeshDeshmukh View on GitHub
Dominant language
Go
Stars
2.2k
Forks
1.1k
Avg merge
3d 22h
Merged PRs (30d)
39

Description

## What happened?
When a user creates a TrainJob with the Flux TrainingRuntime but omits `spec.trainer.numNodes`, the Kubeflow Trainer controller panics with a nil pointer dereference and crash-loops.

In `pkg/runtime/framework/plugins/flux/flux.go`:

- `generateFluxEntrypoint` (line 414) dereferences `*trainJob.Spec.Trainer.NumNodes` without nil checks.
- `generateInitEntrypoint` (line 452) dereferences `*trainJob.Spec.Trainer.NumNodes` without nil checks.
- `EnforceMLPolicy` (lines 149-150) accesses `trainJob.Spec.Trainer.Command` and `trainJob.Spec.Trainer.Args` without nil checks on `Trainer`.

Every other runtime plugin (Torch, JAX, MPI, XGBoost, PlainML) guards with `trainJob.Spec.Trainer != nil && trainJob.Spec.Trainer.NumNodes != nil` before dereferencing.

## What did you expect to happen?
The controller should gracefully handle the missing field by returning an error and marking the TrainJob as Failed, rather than crash-looping.

## Environment
N/A (Codebase bug found in master branch)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.