kubeflow/trainer

Distributed AI Model Training and LLM Fine-Tuning on Kubernetes

Stars: 2.2k
Forks: 1.1k
Open beginner issues: 0
Indexed issues: 90
Avg merge: 3d 22h
Merged PRs (30d): 39

Dominant language: Go
License: Apache-2.0
Last GitHub push: Sep 19, 2026
Latest indexed: Sep 18, 2026
Contributing guide: Contributing guide
Code of conduct: Code of conduct
Beginner labels: help wanted good first issue

Beginner friendly Everything

Open All

90 open issues indexed

[BUG] Distributed environment variables are not injected when TrainJob spec.trainer is omitted in Torch, JAX, and XGBoost plugins Open

kind/bug needs-triage

Difficulty 3/5 1-2 days Newbie friendliness 78/100

kubeflow/trainer#4076 · Opened Sep 17, 2026
MPI: support multiple worker groups with per group `numNodes` and `numProcPerNode` Open

kind/feature needs-triage

kubeflow/trainer#4064 · 3 comments · 2 reactions · 1 assignee · Opened Sep 14, 2026
[BUG] podGroupPolicy accepts both coscheduling and volcano at once Open

kind/bug needs-triage

kubeflow/trainer#4060 · 2 comments · 1 assignee · Opened Sep 12, 2026
[BUG]: IndexableMemTable underflows on an empty batch and fetch_partitions can return a non-overlapping batch Open

kubeflow/trainer#4056 · 1 comment · 1 assignee · Opened Sep 11, 2026
[BUG]: Cache workers with no assigned data files never become ready, so the TrainJob hangs in its init container Open

kubeflow/trainer#4055 · 1 comment · 1 assignee · Opened Sep 11, 2026
bug: nil pointer dereference in Flux plugin when NumProcPerNode is unset Open

Difficulty 2/5 1-3 hours Newbie friendliness 90/100

kubeflow/trainer#4053 · 1 comment · Opened Sep 11, 2026
Implement TTLSecondsAfterFinished and RunPolicy APIs Open

kind/feature needs-triage

Difficulty 5/5 Over a week Newbie friendliness 45/100

kubeflow/trainer#4052 · 3 comments · Opened Sep 10, 2026
[Bug]: Potential nil pointer dereference on NumProcPerNode in MPI plugin EnforceMLPolicy Open

kind/bug needs-triage

Difficulty 2/5 1-3 hours Newbie friendliness 78/100

kubeflow/trainer#4048 · Opened Sep 9, 2026
Volcano PodGroup is not cleaned up when TrainJob transitions to Failed Open

area/scheduler kind/bug

kubeflow/trainer#4033 · 20 comments · 2 assignees · Opened Sep 7, 2026
[KEP-3562] Parallelize TrainJob creation using WorkQueue at scale Open

area/scalability kind/feature

kubeflow/trainer#4031 · 2 comments · 1 assignee · Opened Sep 5, 2026
MPI: `spec.trainer.image` and `spec.trainer.command` cannot both be used once the worker job carries the trainer ancestor label Open

area/mpi kind/bug

kubeflow/trainer#4030 · 6 comments · 1 reaction · 1 assignee · Opened Sep 4, 2026
MPI plugin only sets ORTE env vars, so multi-node jobs silently run single-node on Open MPI 5 Open

area/mpi kind/bug

kubeflow/trainer#4028 · 3 comments · 1 assignee · Opened Sep 4, 2026
bug(runtime): Add support for MPICH and IntelMPI implementations in MPI plugin Open

kind/bug needs-triage

Difficulty 3/5 1-2 days Newbie friendliness 68/100

kubeflow/trainer#4025 · 1 comment · 1 reaction · Opened Sep 2, 2026
Add unit tests for metrics server setup Open

kind/feature needs-triage

kubeflow/trainer#4019 · 2 comments · 1 assignee · Opened Sep 1, 2026
Fix Kubernetes 1.37 integration tests without changing the Trainer API Open

Difficulty 4/5 3-5 days Newbie friendliness 48/100

kubeflow/trainer#4006 · 2 comments · Opened Aug 29, 2026
RuntimeClass handlers ignore TrainingRuntime namespaces when queuing TrainJobs Open

kubeflow/trainer#4001 · 1 comment · 1 assignee · Opened Aug 28, 2026
feat(api): add resources field support to RuntimePatch ContainerPatch API Open

kubeflow/trainer#3999 · 1 comment · 1 assignee · Opened Aug 27, 2026
numGPU returns a nondeterministic value when a container requests multiple GPU-like resources Open

kind/bug

kubeflow/trainer#3996 · 3 comments · 1 assignee · Opened Aug 26, 2026
feat(api): allow spec.trainer.numNodes to be mutable when TrainJob is suspended Open

area/crd kind/enhancement kind/feature

kubeflow/trainer#3995 · 6 comments · 1 assignee · Opened Aug 26, 2026
Initializer environment loading overrides dataclass defaults with None Open

kind/bug

kubeflow/trainer#3993 · 2 comments · 1 assignee · Opened Aug 25, 2026
Runtime reconcilers discard the enriched logger context Open

area/controller kind/bug

kubeflow/trainer#3991 · 1 comment · 1 assignee · Opened Aug 25, 2026
TrainJob can be reconciled against a runtime it was never validated against Open

area/controller

kubeflow/trainer#3975 · 3 comments · 1 assignee · Opened Aug 24, 2026
Status server accepts trailing JSON after a valid status update payload Open

area/controller kind/bug

kubeflow/trainer#3970 · 1 comment · 1 assignee · Opened Aug 24, 2026
JobSet builder discards Info-based Command/Args/Image, breaking the documented EnforceMLPolicyPlugin contract Open

kubeflow/trainer#3967 · 5 comments · 1 assignee · Opened Aug 23, 2026
feat(framework): define deterministic execution order for runtime plugins Open

area/controller kind/feature

kubeflow/trainer#3957 · 7 comments · 1 assignee · Opened Aug 20, 2026
TrainJobStatus silently overwrites conflicting runtime volume or volumeMount Open

kind/bug needs-triage

kubeflow/trainer#3956 · 12 comments · 1 assignee · Opened Aug 20, 2026
Initializer storageUri is silently overridden by user-provided STORAGE_URI env var Open

kind/bug needs-triage

Difficulty 4/5 3-5 days Newbie friendliness 55/100

kubeflow/trainer#3955 · 2 comments · Opened Aug 20, 2026
MPI runLauncherAsNode allocates an extra worker when numNodes is 1 Open

kind/bug needs-triage

Difficulty 3/5 1-2 days Newbie friendliness 57/100

kubeflow/trainer#3954 · 4 comments · Opened Aug 20, 2026
fix(ci): make volcano-crd silently writes a 404 page instead of the PodGroup CRD Open

area/testing kind/bug

kubeflow/trainer#3929 · 6 comments · 1 assignee · Opened Aug 16, 2026
fix(initializer): cache initializer waits for the LeaderWorkerSet with an unbounded poll loop Open

Difficulty 3/5 1-2 days Newbie friendliness 75/100

kubeflow/trainer#3927 · Opened Aug 16, 2026
bug(webhook): Volcano priorityClassName validation exits early, skipping later ReplicatedJobs Open

kubeflow/trainer#3917 · 2 comments · 1 assignee · Opened Aug 14, 2026
feat(initializers): fall back to bin/pt/pth or gguf weights when a HF model repo has no safetensors Open

Difficulty 4/5 3-5 days Newbie friendliness 68/100

kubeflow/trainer#3909 · Opened Aug 13, 2026
bug(api): Int-typed Uniform and LogUniform search spaces accept non-integral bounds Open

area/hpo kind/bug needs-triage

kubeflow/trainer#3906 · 2 comments · 1 assignee · Opened Aug 12, 2026
feat(plugins): validate environment variable collisions across runtime plugins Open

Difficulty 3/5 1-2 days Newbie friendliness 68/100

kubeflow/trainer#3887 · 4 comments · Opened Aug 9, 2026
KEP-3562: [OptimizationJob] Validate Grid search against the search space and trial budget Open

area/hpo kind/feature kind/plan-kep

kubeflow/trainer#3862 · 4 comments · 3 reactions · 1 assignee · Opened Aug 6, 2026
KEP-3562: [OptimizationJob] Implement Shared Initializer for state restoration Open

area/hpo kind/feature kind/plan-kep

kubeflow/trainer#3858 · 7 comments · 1 assignee · Opened Aug 5, 2026
KEP-3562: [OptimizationJob] [TrainJob] Promote TrainJob status feature flag to General Availability (GA) Open

area/hpo kind/feature kind/plan-kep

kubeflow/trainer#3857 · 8 comments · 1 assignee · Opened Aug 5, 2026
Cache initializer silently succeeds on Kubernetes API failures Open

area/data-cache kind/bug

Difficulty 4/5 3-5 days Newbie friendliness 58/100

kubeflow/trainer#3835 · 2 comments · Opened Aug 2, 2026
TrainJob does not retry JobSet deletion after active deadline Open

area/controller kind/bug

Difficulty 3/5 1-2 days Newbie friendliness 68/100

kubeflow/trainer#3833 · 4 comments · Opened Aug 2, 2026
AI Scaffolding: Add .agents skill for add-python-sdk-model Open

kubeflow/trainer#3822 · 1 comment · 1 assignee · Opened Jul 28, 2026
AI Scaffolding: Add .agents skill for add-training-runtime Open

kubeflow/trainer#3821 · 8 comments · 1 assignee · Opened Jul 28, 2026
Add additional panels to Grafana dashboard Open

kubeflow/trainer#3804 · 3 comments · 1 assignee · Opened Jul 25, 2026
Add GPU utilization Grafana dashboard with dcgm-exporter metrics Open

kubeflow/trainer#3803 · 5 comments · 1 assignee · Opened Jul 25, 2026
KEP-3562: [OptimizationJob] Add support for Early Stopping (Pruning), Hyperband, and PBT Open

area/hpo kind/feature kind/plan-kep

kubeflow/trainer#3802 · 8 comments · 1 assignee · Opened Jul 25, 2026
KEP-3562: [OptimizationJob] Add End-to-End (E2E) integration tests for OptimizationJob workflow Open

area/hpo kind/feature

kubeflow/trainer#3801 · 5 comments · 1 assignee · Opened Jul 25, 2026
KEP-3562: [OptimizationJob] Integrate Kueue using `suspend` and `managedBy` fields for queue management Open

area/hpo kind/feature kind/plan-kep

kubeflow/trainer#3800 · 11 comments · 1 assignee · Opened Jul 25, 2026
KEP-3562: [OptimizationJob] Support Multi-Objective Optimization in API and Suggestion Engine Open

area/hpo kind/feature kind/plan-kep

Difficulty 5/5 Over a week Newbie friendliness 45/100

kubeflow/trainer#3799 · 2 comments · Opened Jul 25, 2026
KEP-3562: [OptimizationJob] Implement Secure Auth between Controller and gRPC Suggestion Service Open

area/hpo kind/feature kind/plan-kep

kubeflow/trainer#3798 · 3 comments · 1 assignee · Opened Jul 25, 2026
KEP-3562: [OptimizationJob] Implement and refactor Optuna suggestion service backend Open

area/hpo kind/feature kind/plan-kep

kubeflow/trainer#3797 · 3 comments · 1 assignee · Opened Jul 25, 2026
KEP-3562: [OptimizationJob] Refactor Katib gRPC protocol contracts for stateless suggestion services Open

area/hpo kind/feature kind/plan-kep

kubeflow/trainer#3796 · 7 comments · 1 assignee · Opened Jul 25, 2026

A short digest of beginner-friendly GitHub issues.