fix(initializer): cache initializer waits for the LeaderWorkerSet with an unbounded poll loop
- Dominant language
- Go
- Stars
- 2.2k
- Forks
- 1.1k
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 39
Description
The cache initializer waits for the LeaderWorkerSet to come up with a poll loop, and there's a TODO on it asking for the watch API:
https://github.com/kubeflow/trainer/blob/master/pkg/initializers/dataset/cache.py#L309-L333
```python
# Wait for LeaderWorkerSet to become ready
# TODO:// refactor to use watch API
while True:
...
time.sleep(5)
```
Two problems with it as written.
The loop has no exit other than success. If the LeaderWorkerSet never reports `Available` (bad image, unschedulable pods, missing quota) the initializer sits in `time.sleep(5)` forever. The init container never finishes, the TrainJob never starts, and nothing in the logs says why beyond the last message before the loop. Right now the only way out is deleting the TrainJob.
The polling itself is also wasteful: a GET every 5s per cache-backed TrainJob for the whole startup window, when the API server can just push the change.
Switching to `watch.Watch().stream(...)` fixes both, since `timeout_seconds` gives the bounded wait for free. The watch sends the current object as an ADDED event before any updates, so a cluster that's already up is still picked up immediately.
The wait needs its own config value. `readiness_timeout_seconds` already exists but it's the per-probe timeout on the container readinessProbe (default 5s), so it can't be reused for the overall wait.
Scope:
- `pkg/initializers/dataset/cache.py` — replace the poll with a watch, raise on timeout
- `pkg/initializers/types/types.py` — add `cache_ready_timeout_seconds`
- `pkg/initializers/dataset/cache_test.py` — the two existing tests drive readiness through `get_namespaced_custom_object.side_effect` and need reworking, plus cases for the timeout path
Happy to pick this one up.
Contributor guide
Research direction
Start in pkg/initializers/dataset/cache.py at the LeaderWorkerSet readiness poll, then review the related configuration in pkg/initializers/types/types.py. Run pkg/initializers/dataset/cache_test.py and rework the existing readiness tests around the watch behavior. Done means readiness is observed through the watch, an existing object is handled, and timeout behavior is covered by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, python
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 75/100