kubeflow / kubeflow/trainer

[BUG] podGroupPolicy accepts both coscheduling and volcano at once

Open
#4,060 2 comments 0 reactions 1 assignee Claimed by @rishabhsai View on GitHub
kind/bug needs-triage
Dominant language
Go
Stars
2.2k
Forks
1.1k
Avg merge
3d 22h
Merged PRs (30d)
39

Description

### What happened?

`PodGroupPolicySource` allows both `coscheduling` and `volcano` to be specified simultaneously, even though the API documentation states that only one of them may be specified.

The adjacent `MLPolicy` type already enforces mutual exclusivity using CEL validation, but `PodGroupPolicySource` has no equivalent validation.

For example, the following configuration is currently accepted:

```yaml
spec:
podGroupPolicy:
coscheduling:
scheduleTimeoutSeconds: 30
volcano: {}
```

Both the Volcano and CoScheduling plugins then activate for the same TrainJob. Each plugin creates its own PodGroup in its respective API group and applies its scheduling identity to the pods.

This can leave the TrainJob stuck in Pending without an admission error or actionable condition indicating that both gang-scheduling policies were configured.

The generated CRDs also do not contain an x-kubernetes-validations rule under podGroupPolicy, and the runtime webhooks do not appear to validate this exclusivity.

### What did you expect to happen?

```md
The API should reject a `TrainingRuntime` or `ClusterTrainingRuntime` when both `podGroupPolicy.coscheduling` and `podGroupPolicy.volcano` are specified.

This should preferably be enforced at admission time using a CEL validation rule, following the existing mutual-exclusivity validation used by `MLPolicy`.

A clear validation error should indicate that only one of `coscheduling` or `volcano` may be specified.
```

### Environment

- Kubeflow Trainer: upstream `master` @ `8ca43ecb`
- Kubernetes: Kubernetes cluster with Kubeflow Trainer installed
- Resource: `TrainingRuntime` / `ClusterTrainingRuntime`
- Affected area: `PodGroupPolicy` / Volcano and CoScheduling plugins
- Installation: Kubernetes deployment

### Impacted by this bug?

Give it a 👍 We prioritize the issues with most 👍

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.