[Bug]: container image push fails to register manifest digests referenced in OCI Index, causing 404s on pull
- Dominant language
- Swift
- Stars
- 49.9k
- Forks
- 1.8k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 22
Description
### I have done the following
- [x] I have searched the existing issues
- [ ] If possible, I've reproduced the issue using the 'main' branch of this project
### Steps to reproduce
# Reproduction Steps:
1. Build for a specific arch: `container build --arch amd64 -t my-reg/my-image:v1`
2. Push to registry: `container image push my-reg/my-image:v1`
3. Attempt to pull from another machine: `crictl pull my-reg/my-image:v1`
# Observed Registry Logs (Zot):
```
{"method":"GET","path":"/v2/my-image/manifests/sha256:","statusCode":200}
{"method":"GET","path":"/v2/my-image/manifests/sha256:","statusCode":404}
```
# Observed K8s Events:
- Failed to pull image "...": rpc error: code = NotFound desc = failed to pull and unpack image "...":
- failed to copy: httpReadSeeker: failed open: content at .../manifests/sha256: not found
# Workaround:
Exporting the image to a tarball and using `skopeo copy` to the registry fixes the issue, as `skopeo` correctly handles the manifest registration:
```
container image save my-reg/my-image:v1 -o image.tar
skopeo copy oci-archive:image.tar docker://my-reg/my-image:v1
```
### Current behavior
When building an image for a specific architecture (e.g., --arch amd64) and pushing it to a registry, the container CLI creates and pushes an OCI Image Index (manifest
list). While the index itself is pushed successfully, the platform-specific manifest it references is not being correctly "registered" in the registry's manifest store.
When a client (like containerd) attempts to pull the image:
1. It successfully fetches the OCI Index by tag.
2. It identifies the correct manifest digest for the target architecture from the index.
3. It attempts to fetch that specific manifest by digest: GET /v2//manifests/sha256:.
4. The registry returns 404 Not Found.
Investigation shows the manifest content was uploaded as a blob, but the registry does not recognize it as a valid manifest endpoint, suggesting the push command is
missing the final PUT to the manifests functional endpoint for the sub-manifests.
### Expected behavior
The push command should ensure all manifests referenced by the OCI Index are correctly registered at the registry's /manifests/ endpoint so they can be pulled by
content-addressable digest.
### Environment
```markdown
- OS: macOS Version 15.7.3 (24G419)
- Container: container CLI version 0.7.1 (build: release, commit: 420be74)
```
### Relevant log output
```shell
N/A
```
### Code of Conduct
- [x] I agree to follow this project's Code of Conduct
Contributor guide
Research direction
Start by tracing the `container image push` command and its OCI Index handling, then compare the push sequence with the documented `/manifests/` endpoint behavior and the working `skopeo copy` workaround. Done means the platform-specific manifests referenced by the index can be pulled by digest from the registry, including through `crictl`.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- swift
- Domain
- api, cli
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100