flyteorg / flyteorg/flyte

[BUG] Unable to launch any task as flytekit image will not pull down

Open
#6,498 2 comments 0 reactions 0 assignees View on GitHub
bug untriaged
Dominant language
Go
Stars
7.5k
Forks
886
Avg merge
1d 14h
Merged PRs (30d)
120

Description

### Flyte & Flytekit version

Fresh install of sandbox. But also impacting our environments pinned to version v1.13.3.

flytekit:py3.11-1.16.0

### Describe the bug

Any new pods without the cached flytekit image fail to run with the bug below.

start failed in pod ashkglxk5xwsvc8kfk8x-n0-0_flytesnacks-development(c1cf7366-1f53-42c0-bc7b-bfd9962c8016): ErrImagePull: rpc error: code = NotFound desc = failed to pull and unpack image "[cr.flyte.org/flyteorg/flytekit:py3.11-1.16.0](http://cr.flyte.org/flyteorg/flytekit:py3.11-1.16.0)": failed to resolve reference "[cr.flyte.org/flyteorg/flytekit:py3.11-1.16.0](http://cr.flyte.org/flyteorg/flytekit:py3.11-1.16.0)": [cr.flyte.org/flyteorg/flytekit:py3.11-1.16.0](http://cr.flyte.org/flyteorg/flytekit:py3.11-1.16.0): not found
E0609 13:08:48.802517

When looking at the flytekit versions all old versions are tagged in the form of:
Py{version}-{flytekit-version} [py3.11-1.14.8]. (https://github.com/orgs/flyteorg/packages/container/flytekit/400696278?tag=py3.11-1.14.8)
[py3.11-9188d1b497abd67944e064933df51394b1b9cfe5](https://github.com/orgs/flyteorg/packages/container/flytekit/400696278?tag=py3.11-9188d1b497abd67944e064933df51394b1b9cfe5)

but the newest appended a v in its tag:
py{version}-v{flytekit-version} [py3.11-v1.16.0](https://github.com/orgs/flyteorg/packages/container/flytekit/427696580?tag=py3.11-v1.16.0)
[py3.11-764b36f70b05bb01b1bfad5d75c48bdb2b4bcc35](https://github.com/orgs/flyteorg/packages/container/flytekit/427696580?tag=py3.11-764b36f70b05bb01b1bfad5d75c48bdb2b4bcc35)

### Expected behavior

When a new pod pulls the current flytekit there is a successful download and the pod begins running after download.

### Additional context to reproduce

_No response_

### Screenshots

_No response_

### Are you sure this issue hasn't been raised already?

- [x] Yes

### Have you read the Code of Conduct?

- [x] Yes

Contributor guide

Open the contributing guide

Research direction

Start by comparing the failing image reference, cr.flyte.org/flyteorg/flytekit:py3.11-1.16.0, with the published py3.11-v1.16.0 and commit-based tags linked in the issue. Trace the flytekit image publication or tag-generation entry point, then reproduce a new pod launch and confirm that the expected image tag pulls successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, kubernetes, python
Domain
devops, infrastructure, release
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.