kubeflow / kubeflow/trainer

fix(initializer): cache initializer waits for the LeaderWorkerSet with an unbounded poll loop

Open
#3,927 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.2k
Forks
1.1k
Avg merge
3d 22h
Merged PRs (30d)
39

Description

The cache initializer waits for the LeaderWorkerSet to come up with a poll loop, and there's a TODO on it asking for the watch API:

https://github.com/kubeflow/trainer/blob/master/pkg/initializers/dataset/cache.py#L309-L333

```python
# Wait for LeaderWorkerSet to become ready
# TODO:// refactor to use watch API
while True:
...
time.sleep(5)
```

Two problems with it as written.

The loop has no exit other than success. If the LeaderWorkerSet never reports `Available` (bad image, unschedulable pods, missing quota) the initializer sits in `time.sleep(5)` forever. The init container never finishes, the TrainJob never starts, and nothing in the logs says why beyond the last message before the loop. Right now the only way out is deleting the TrainJob.

The polling itself is also wasteful: a GET every 5s per cache-backed TrainJob for the whole startup window, when the API server can just push the change.

Switching to `watch.Watch().stream(...)` fixes both, since `timeout_seconds` gives the bounded wait for free. The watch sends the current object as an ADDED event before any updates, so a cluster that's already up is still picked up immediately.

The wait needs its own config value. `readiness_timeout_seconds` already exists but it's the per-probe timeout on the container readinessProbe (default 5s), so it can't be reused for the overall wait.

Scope:
- `pkg/initializers/dataset/cache.py` — replace the poll with a watch, raise on timeout
- `pkg/initializers/types/types.py` — add `cache_ready_timeout_seconds`
- `pkg/initializers/dataset/cache_test.py` — the two existing tests drive readiness through `get_namespaced_custom_object.side_effect` and need reworking, plus cases for the timeout path

Happy to pick this one up.

Contributor guide

Open the contributing guide

Research direction

Start in pkg/initializers/dataset/cache.py at the LeaderWorkerSet readiness poll, then review the related configuration in pkg/initializers/types/types.py. Run pkg/initializers/dataset/cache_test.py and rework the existing readiness tests around the watch behavior. Done means readiness is observed through the watch, an existing object is handled, and timeout behavior is covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, python
Domain
devops, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
75/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.