apple / apple/container

[Bug]: container image push to ECR fails with 416 BLOB_UPLOAD_INVALID after partial blob upload retry

Open
#1,895 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Swift
Stars
49.9k
Forks
1.8k
Avg merge
1d 20h
Merged PRs (30d)
22

Description

### I have done the following

- [x] I have searched the existing issues
- [x] If possible, I've reproduced the issue using the 'main' branch of this project

### Summary

`container image push` to **AWS ECR** fails mid-push with HTTP **416 Range Not Satisfiable** and ECR error code **`BLOB_UPLOAD_INVALID`**. The registry reports that the client restarted a blob upload from byte **0** while an in-progress upload session already had bytes committed (example from one failure: **11,304,960**).

This appears related to blob upload retry/resume logic in the **`containerization`** dependency (`RegistryClient+Push.swift`), not the thin `container` CLI wrapper.

**Related but distinct open issues:**
- #1707 — ECR manifest PUT returns 401 after successful blob upload
- #1001 — OCI index child manifests not registered on push (404 on pull)
- #1310 — private registry push / keychain failures

No existing issue was found for **`BLOB_UPLOAD_INVALID`** / **416** / **"First byte of the layer part"** on ECR.

---

### Steps to reproduce

```sh
# 1. Login to ECR
aws ecr get-login-password --region \
| container registry login --username AWS --password-stdin \
.dkr.ecr..amazonaws.com

# 2. Build a multi-arch image locally
container build \
--platform linux/amd64,linux/arm64 \
-t .dkr.ecr..amazonaws.com/: \
.

# 3. Push to ECR
container image push .dkr.ecr..amazonaws.com/:
```

The issue is most likely to appear with **large multi-arch images** when a blob upload is interrupted or retried (network error, auth refresh, or re-running push after a partial failure). Smaller or single-arch images may push successfully on the first attempt.

---

### Current behavior

Push progresses through blob uploads, then fails with an error similar to:

```
Error: HTTP request to https://.dkr.ecr..amazonaws.com/v2//blobs/uploads/?digest=sha256: failed with response: 416 Range Not Satisfiable. Reason: {"errors":[{"code":"BLOB_UPLOAD_INVALID","message":"First byte of the layer part is 0 instead of 11304960"}]}
```

Typical pattern:

1. Push starts and uploads some blobs successfully.
2. A large blob upload is partially sent, then the request fails or is retried.
3. The client retries the **same upload session URL** but restarts the payload from **byte 0**.
4. ECR rejects the request with **`BLOB_UPLOAD_INVALID`** (HTTP 416).

The local image build itself can succeed; the failure occurs during **registry push**.

---

### Expected behavior

- Blob uploads should either **resume** from the last committed byte on an existing upload session, or **abandon** the stale session and start a fresh `POST /v2//blobs/uploads/` before retrying.
- Retrying a failed or interrupted push should complete successfully without manual intervention.

---

### Root cause analysis (suspected)

The push implementation lives in **`apple/containerization`**, not `apple/container`:

**`Sources/ContainerizationOCI/Client/RegistryClient+Push.swift`**

1. `POST /v2//blobs/uploads/` → registry returns upload URL with session UUID
2. Single monolithic `PUT` of entire blob to `.../blobs/uploads/?digest=sha256:...`
3. Comment: *"We have to pass a body closure rather than a body to reset the stream when retrying."*

**`Sources/ContainerizationOCI/Client/RegistryClient.swift`**

The generic `request()` loop retries failed requests by calling `bodyClosure()` again (stream resets to **byte 0**) while reusing the **same upload URL**:

```swift
while true {
request.body = try bodyClosure() // stream restarts at byte 0
let _response = try await client.execute(request, ...)
// retries on 5xx (shouldRetry) and network errors in catch block
continue // same URL, fresh stream from 0
}
```

If the first `PUT` partially uploads then fails or is retried, ECR expects the next request to continue from the last committed byte, but the client sends from byte **0** → **`BLOB_UPLOAD_INVALID`**.

This matches known behavior in other OCI clients:
- [google/go-containerregistry#431](https://github.com/google/go-containerregistry/issues/431) — Handling `BLOB_UPLOAD_INVALID`
- [GoogleContainerTools/kaniko#607](https://github.com/GoogleContainerTools/kaniko/issues/607) — `BLOB_UPLOAD_INVALID` when pushing to ECR

**Suggested fix (in `containerization`):**

1. Wrap blob push in an outer retry that always issues a **new `POST`** (new upload session) on `BLOB_UPLOAD_INVALID` / 416.
2. Do **not** retry blob `PUT` via generic `request()` retry without a new upload session.
3. (Longer term) Implement distribution-spec **PATCH** uploads with `Content-Range` and `GET` upload status for resume.

---

### Workaround

Push via another client after saving locally:

```sh
container image save -o image.tar .dkr.ecr..amazonaws.com/:
docker load < image.tar
aws ecr get-login-password --region \
| docker login --username AWS --password-stdin .dkr.ecr..amazonaws.com
docker push .dkr.ecr..amazonaws.com/:
```

Also suggested in #1001: `container image save` + `skopeo copy oci-archive:... docker://...`

---

### Environment

```markdown
- OS: macOS 26.5.1
- Container CLI: version 1.0.0 (build: release, commit: ee848e3)
- Registry: AWS ECR
- Image: multi-arch OCI index (linux/amd64 + linux/arm64)
- Auth: `container registry login` with ECR password from `aws ecr get-login-password`
```

---

### Code of Conduct

- [x] I agree to follow this project's Code of Conduct

Contributor guide

Open the contributing guide

Research direction

Start with Sources/ContainerizationOCI/Client/RegistryClient+Push.swift and Sources/ContainerizationOCI/Client/RegistryClient.swift, focusing on the blob upload body closure and generic retry loop. Reproduce an interrupted large multi-arch push to ECR and verify that retrying completes without HTTP 416 or BLOB_UPLOAD_INVALID, either by resuming correctly or using a fresh upload session.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, swift
Domain
backend-api-design, cloud, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.