apache / apache/texera

Pre-pull curated images onto every node

Closed
#8,469 1 comment 0 reactions 1 assignee Claimed by @tanishqgandhi1908 View on GitHub
Dominant language
Scala
Stars
314
Forks
187
Avg merge
1d 21h
Merged PRs (30d)
214

Description

### Task Summary

The first unit on a node waits for the whole image — about 80 seconds for a 3 GB one — while every later unit there starts at once. The same action takes seconds or minutes depending on which node it lands on.

Pull each ready image onto every node as soon as it is ready, rather than when someone first starts a unit from it.

Same mechanism Texera already uses for the default computing-unit image: a DaemonSet whose init container is the image and whose command does nothing, then a small pause container to hold the pod open so the image is not reclaimed. One per image, removed when the image is.

If it cannot start it is logged and ignored — the image still works, the first unit just pays for the pull.

Costs node disk: every node holds every ready image. Configurable.

Done when a ready image is on every node before anyone starts a unit from it, and deleting the image removes its pre-pull.

Part of #8466

### Task Type

- [ ] Refactor / Cleanup
- [ ] DevOps / Deployment / CI
- [ ] Testing / QA
- [ ] Documentation
- [ ] Performance
- [ ] Other

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.