[BUG] Unable to launch any task as flytekit image will not pull down
- Dominant language
- Go
- Stars
- 7.5k
- Forks
- 886
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 120
Description
### Flyte & Flytekit version
Fresh install of sandbox. But also impacting our environments pinned to version v1.13.3.
flytekit:py3.11-1.16.0
### Describe the bug
Any new pods without the cached flytekit image fail to run with the bug below.
start failed in pod ashkglxk5xwsvc8kfk8x-n0-0_flytesnacks-development(c1cf7366-1f53-42c0-bc7b-bfd9962c8016): ErrImagePull: rpc error: code = NotFound desc = failed to pull and unpack image "[cr.flyte.org/flyteorg/flytekit:py3.11-1.16.0](http://cr.flyte.org/flyteorg/flytekit:py3.11-1.16.0)": failed to resolve reference "[cr.flyte.org/flyteorg/flytekit:py3.11-1.16.0](http://cr.flyte.org/flyteorg/flytekit:py3.11-1.16.0)": [cr.flyte.org/flyteorg/flytekit:py3.11-1.16.0](http://cr.flyte.org/flyteorg/flytekit:py3.11-1.16.0): not found
E0609 13:08:48.802517
When looking at the flytekit versions all old versions are tagged in the form of:
Py{version}-{flytekit-version} [py3.11-1.14.8]. (https://github.com/orgs/flyteorg/packages/container/flytekit/400696278?tag=py3.11-1.14.8)
[py3.11-9188d1b497abd67944e064933df51394b1b9cfe5](https://github.com/orgs/flyteorg/packages/container/flytekit/400696278?tag=py3.11-9188d1b497abd67944e064933df51394b1b9cfe5)
but the newest appended a v in its tag:
py{version}-v{flytekit-version} [py3.11-v1.16.0](https://github.com/orgs/flyteorg/packages/container/flytekit/427696580?tag=py3.11-v1.16.0)
[py3.11-764b36f70b05bb01b1bfad5d75c48bdb2b4bcc35](https://github.com/orgs/flyteorg/packages/container/flytekit/427696580?tag=py3.11-764b36f70b05bb01b1bfad5d75c48bdb2b4bcc35)
### Expected behavior
When a new pod pulls the current flytekit there is a successful download and the pod begins running after download.
### Additional context to reproduce
_No response_
### Screenshots
_No response_
### Are you sure this issue hasn't been raised already?
- [x] Yes
### Have you read the Code of Conduct?
- [x] Yes
Contributor guide
Research direction
Start by comparing the failing image reference, cr.flyte.org/flyteorg/flytekit:py3.11-1.16.0, with the published py3.11-v1.16.0 and commit-based tags linked in the issue. Trace the flytekit image publication or tag-generation entry point, then reproduce a new pod launch and confirm that the expected image tag pulls successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, kubernetes, python
- Domain
- devops, infrastructure, release
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100