[Bug]: container image push to ECR fails with 416 BLOB_UPLOAD_INVALID after partial blob upload retry
- Dominant language
- Swift
- Stars
- 49.9k
- Forks
- 1.8k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 22
Description
### I have done the following
- [x] I have searched the existing issues
- [x] If possible, I've reproduced the issue using the 'main' branch of this project
### Summary
`container image push` to **AWS ECR** fails mid-push with HTTP **416 Range Not Satisfiable** and ECR error code **`BLOB_UPLOAD_INVALID`**. The registry reports that the client restarted a blob upload from byte **0** while an in-progress upload session already had bytes committed (example from one failure: **11,304,960**).
This appears related to blob upload retry/resume logic in the **`containerization`** dependency (`RegistryClient+Push.swift`), not the thin `container` CLI wrapper.
**Related but distinct open issues:**
- #1707 — ECR manifest PUT returns 401 after successful blob upload
- #1001 — OCI index child manifests not registered on push (404 on pull)
- #1310 — private registry push / keychain failures
No existing issue was found for **`BLOB_UPLOAD_INVALID`** / **416** / **"First byte of the layer part"** on ECR.
---
### Steps to reproduce
```sh
# 1. Login to ECR
aws ecr get-login-password --region \
| container registry login --username AWS --password-stdin \
.dkr.ecr..amazonaws.com
# 2. Build a multi-arch image locally
container build \
--platform linux/amd64,linux/arm64 \
-t .dkr.ecr..amazonaws.com/: \
.
# 3. Push to ECR
container image push .dkr.ecr..amazonaws.com/:
```
The issue is most likely to appear with **large multi-arch images** when a blob upload is interrupted or retried (network error, auth refresh, or re-running push after a partial failure). Smaller or single-arch images may push successfully on the first attempt.
---
### Current behavior
Push progresses through blob uploads, then fails with an error similar to:
```
Error: HTTP request to https://.dkr.ecr..amazonaws.com/v2//blobs/uploads/?digest=sha256: failed with response: 416 Range Not Satisfiable. Reason: {"errors":[{"code":"BLOB_UPLOAD_INVALID","message":"First byte of the layer part is 0 instead of 11304960"}]}
```
Typical pattern:
1. Push starts and uploads some blobs successfully.
2. A large blob upload is partially sent, then the request fails or is retried.
3. The client retries the **same upload session URL** but restarts the payload from **byte 0**.
4. ECR rejects the request with **`BLOB_UPLOAD_INVALID`** (HTTP 416).
The local image build itself can succeed; the failure occurs during **registry push**.
---
### Expected behavior
- Blob uploads should either **resume** from the last committed byte on an existing upload session, or **abandon** the stale session and start a fresh `POST /v2//blobs/uploads/` before retrying.
- Retrying a failed or interrupted push should complete successfully without manual intervention.
---
### Root cause analysis (suspected)
The push implementation lives in **`apple/containerization`**, not `apple/container`:
**`Sources/ContainerizationOCI/Client/RegistryClient+Push.swift`**
1. `POST /v2//blobs/uploads/` → registry returns upload URL with session UUID
2. Single monolithic `PUT` of entire blob to `.../blobs/uploads/?digest=sha256:...`
3. Comment: *"We have to pass a body closure rather than a body to reset the stream when retrying."*
**`Sources/ContainerizationOCI/Client/RegistryClient.swift`**
The generic `request()` loop retries failed requests by calling `bodyClosure()` again (stream resets to **byte 0**) while reusing the **same upload URL**:
```swift
while true {
request.body = try bodyClosure() // stream restarts at byte 0
let _response = try await client.execute(request, ...)
// retries on 5xx (shouldRetry) and network errors in catch block
continue // same URL, fresh stream from 0
}
```
If the first `PUT` partially uploads then fails or is retried, ECR expects the next request to continue from the last committed byte, but the client sends from byte **0** → **`BLOB_UPLOAD_INVALID`**.
This matches known behavior in other OCI clients:
- [google/go-containerregistry#431](https://github.com/google/go-containerregistry/issues/431) — Handling `BLOB_UPLOAD_INVALID`
- [GoogleContainerTools/kaniko#607](https://github.com/GoogleContainerTools/kaniko/issues/607) — `BLOB_UPLOAD_INVALID` when pushing to ECR
**Suggested fix (in `containerization`):**
1. Wrap blob push in an outer retry that always issues a **new `POST`** (new upload session) on `BLOB_UPLOAD_INVALID` / 416.
2. Do **not** retry blob `PUT` via generic `request()` retry without a new upload session.
3. (Longer term) Implement distribution-spec **PATCH** uploads with `Content-Range` and `GET` upload status for resume.
---
### Workaround
Push via another client after saving locally:
```sh
container image save -o image.tar .dkr.ecr..amazonaws.com/:
docker load < image.tar
aws ecr get-login-password --region \
| docker login --username AWS --password-stdin .dkr.ecr..amazonaws.com
docker push .dkr.ecr..amazonaws.com/:
```
Also suggested in #1001: `container image save` + `skopeo copy oci-archive:... docker://...`
---
### Environment
```markdown
- OS: macOS 26.5.1
- Container CLI: version 1.0.0 (build: release, commit: ee848e3)
- Registry: AWS ECR
- Image: multi-arch OCI index (linux/amd64 + linux/arm64)
- Auth: `container registry login` with ECR password from `aws ecr get-login-password`
```
---
### Code of Conduct
- [x] I agree to follow this project's Code of Conduct
Contributor guide
Research direction
Start with Sources/ContainerizationOCI/Client/RegistryClient+Push.swift and Sources/ContainerizationOCI/Client/RegistryClient.swift, focusing on the blob upload body closure and generic retry loop. Reproduce an interrupted large multi-arch push to ECR and verify that retrying completes without HTTP 416 or BLOB_UPLOAD_INVALID, either by resuming correctly or using a fresh upload session.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, swift
- Domain
- backend-api-design, cloud, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100