apple / apple/container

[Bug]: container image push fails to register manifest digests referenced in OCI Index, causing 404s on pull

Open
#1,001 1 comment 2 reactions 0 assignees View on GitHub
Dominant language
Swift
Stars
49.9k
Forks
1.8k
Avg merge
1d 20h
Merged PRs (30d)
22

Description

### I have done the following

- [x] I have searched the existing issues
- [ ] If possible, I've reproduced the issue using the 'main' branch of this project

### Steps to reproduce

# Reproduction Steps:
1. Build for a specific arch: `container build --arch amd64 -t my-reg/my-image:v1`
2. Push to registry: `container image push my-reg/my-image:v1`
3. Attempt to pull from another machine: `crictl pull my-reg/my-image:v1`

# Observed Registry Logs (Zot):

```
{"method":"GET","path":"/v2/my-image/manifests/sha256:","statusCode":200}
{"method":"GET","path":"/v2/my-image/manifests/sha256:","statusCode":404}
```

# Observed K8s Events:

- Failed to pull image "...": rpc error: code = NotFound desc = failed to pull and unpack image "...":
- failed to copy: httpReadSeeker: failed open: content at .../manifests/sha256: not found

# Workaround:
Exporting the image to a tarball and using `skopeo copy` to the registry fixes the issue, as `skopeo` correctly handles the manifest registration:

```
container image save my-reg/my-image:v1 -o image.tar
skopeo copy oci-archive:image.tar docker://my-reg/my-image:v1
```

### Current behavior

When building an image for a specific architecture (e.g., --arch amd64) and pushing it to a registry, the container CLI creates and pushes an OCI Image Index (manifest
list). While the index itself is pushed successfully, the platform-specific manifest it references is not being correctly "registered" in the registry's manifest store.

When a client (like containerd) attempts to pull the image:
1. It successfully fetches the OCI Index by tag.
2. It identifies the correct manifest digest for the target architecture from the index.
3. It attempts to fetch that specific manifest by digest: GET /v2//manifests/sha256:.
4. The registry returns 404 Not Found.

Investigation shows the manifest content was uploaded as a blob, but the registry does not recognize it as a valid manifest endpoint, suggesting the push command is
missing the final PUT to the manifests functional endpoint for the sub-manifests.

### Expected behavior

The push command should ensure all manifests referenced by the OCI Index are correctly registered at the registry's /manifests/ endpoint so they can be pulled by
content-addressable digest.

### Environment

```markdown
- OS: macOS Version 15.7.3 (24G419)
- Container: container CLI version 0.7.1 (build: release, commit: 420be74)
```

### Relevant log output

```shell
N/A
```

### Code of Conduct

- [x] I agree to follow this project's Code of Conduct

Contributor guide

Open the contributing guide

Research direction

Start by tracing the `container image push` command and its OCI Index handling, then compare the push sequence with the documented `/manifests/` endpoint behavior and the working `skopeo copy` workaround. Done means the platform-specific manifests referenced by the index can be pulled by digest from the registry, including through `crictl`.

Written by the indexing model from the issue text.

Assessment

Tech stack
swift
Domain
api, cli
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.