kubeflow / kubeflow/trainer

feat(plugins): validate environment variable collisions across runtime plugins

Open
#3,887 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.2k
Forks
1.1k
Avg merge
3d 22h
Merged PRs (30d)
39

Description

Currently, when multiple runtime plugins (e.g., `Torch`, `JobSet`, `Volcano`, `TrainJobStatus`) execute during the runtime reconciliation pipeline, environment variables are injected into container specifications via `apply.UpsertEnvVars`.
However, there is no validation step to detect if two plugins attempt to inject conflicting or colliding environment variable names with different values into the same container.

Reference TODO comment in `torch.go`:
TODO (andreyvelich): We should validate that envs from different plugins don't conflict with each other.
Ref: https://github.com/kubeflow/trainer/pull/2308#discussion_r1823229940

Proposed Solution -
Introduce a validation check in pkg/runtime/framework/core during plugin pipeline execution to track injected environment variable keys per container.
Emit an error or validation warning when an environment variable key is overridden by a subsequent plugin with a conflicting value.
Add unit test coverage in pkg/runtime/framework/core/framework_test.go.
Target Files to Change
pkg/runtime/framework/plugins/torch/torch.go (line 171)
pkg/runtime/framework/core/framework.go
pkg/runtime/framework/core/framework_test.go

Assignment Request -
I would like to work on this issue. Please assign it to me!

Contributor guide

Open the contributing guide

Research direction

Start in pkg/runtime/framework/core/framework.go by tracing plugin pipeline execution and read the existing tests in pkg/runtime/framework/core/framework_test.go. Review the TODO at pkg/runtime/framework/plugins/torch/torch.go:171 and the existing apply.UpsertEnvVars behavior. Done means conflicting environment variable injections are detected during execution and covered by unit tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend, infrastructure
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.