apple / apple/containerization
[Bug]: RegistryClient blob push retry restarts upload from byte 0, causing ECR BLOB_UPLOAD_INVALID (416)
- Dominant language
- Swift
- Stars
- 8.9k
- Forks
- 359
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 13
Description
### Summary
`RegistryClient` blob push to **AWS ECR** can fail with HTTP **416** and error code **`BLOB_UPLOAD_INVALID`** when a large blob upload is interrupted or retried. ECR reports that the client restarted from byte **0** while the upload session already had bytes committed.
This is the underlying implementation issue tracked upstream in the **`container`** CLI repo: **apple/container#1895**
---
### Steps to reproduce
Reproducible via `container image push` (which uses this library). Minimal flow:
```sh
aws ecr get-login-password --region \
| container registry login --username AWS --password-stdin \
.dkr.ecr..amazonaws.com
container build \
--platform linux/amd64,linux/arm64 \
-t .dkr.ecr..amazonaws.com/: \
.
container image push .dkr.ecr..amazonaws.com/:
```
Most likely with **large multi-arch images** when a blob upload is partially sent then retried.
---
### Current behavior
Example error:
```
416 Range Not Satisfiable
{"errors":[{"code":"BLOB_UPLOAD_INVALID","message":"First byte of the layer part is 0 instead of 11304960"}]}
```
---
### Suspected root cause
**`Sources/ContainerizationOCI/Client/RegistryClient+Push.swift`**
Blob push flow:
1. `POST /v2//blobs/uploads/` → obtain upload session URL
2. Monolithic `PUT` of entire blob to `.../blobs/uploads/?digest=sha256:...`
The code passes a `bodyClosure` that resets the stream on retry:
```swift
// We have to pass a body closure rather than a body to reset the stream when retrying.
let bodyClosure = {
let stream = try streamGenerator()
return HTTPClientRequest.Body.stream(stream, length: .known(descriptor.size))
}
try await request(components: components, method: .PUT, bodyClosure: bodyClosure, ...)
```
**`Sources/ContainerizationOCI/Client/RegistryClient.swift`**
The generic `request()` loop retries by re-invoking `bodyClosure()` (stream from byte **0**) against the **same request URL**:
```swift
while true {
request.body = try bodyClosure()
let _response = try await client.execute(request, ...)
// retries on 5xx (shouldRetry) and network errors
continue
}
```
For blob uploads, if `PUT` partially succeeds then retries (network error or generic retry), the registry expects continuation from the last committed byte, not a restart from byte 0.
Similar issues in other OCI clients:
- [google/go-containerregistry#431](https://github.com/google/go-containerregistry/issues/431)
- [GoogleContainerTools/kaniko#607](https://github.com/GoogleContainerTools/kaniko/issues/607)
---
### Expected behavior
On blob upload failure or `BLOB_UPLOAD_INVALID`:
1. **Abandon** the stale upload session and issue a fresh `POST /blobs/uploads/`, then retry the full blob; **or**
2. **Resume** via distribution-spec chunked upload (`GET` upload status + `PATCH` with `Content-Range`).
Generic `request()` retry must not restart a blob `PUT` from byte 0 on the same upload session URL.
---
### Suggested fix
1. Refactor `push()` in `RegistryClient+Push.swift` to separate `initiateBlobUpload()` and `uploadBlob()`.
2. Wrap blob upload in an outer retry loop; on `BLOB_UPLOAD_INVALID` / HTTP 416, always start a **new upload session** (`POST`).
3. Disable or bypass generic `request()` retry for blob `PUT`, or pass `retryOptions: nil` for that call.
4. Add tests in `Tests/ContainerizationOCITests/` simulating partial upload + 416 → successful retry with new session.
5. (Optional) Implement `PATCH` + `Content-Range` resume per the [distribution spec](https://distribution.github.io/distribution/spec/api/#pushing-a-layer).
---
### Related
- **Upstream report (container CLI):** https://github.com/apple/container/issues/1895
- **Related container issues:** apple/container#1707 (ECR manifest PUT 401), apple/container#1001 (OCI index manifest registration)
Contributor guide
Research direction
Read Sources/ContainerizationOCI/Client/RegistryClient+Push.swift and RegistryClient.swift to trace blob upload retries and body stream recreation. Then inspect Tests/ContainerizationOCITests/ for registry test patterns and add a partial-upload HTTP 416 case. Done means a failed blob upload abandons the stale session and retries through a new session without restarting on the same URL.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, swift
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 50/100