apple / apple/containerization

[Bug]: RegistryClient blob push retry restarts upload from byte 0, causing ECR BLOB_UPLOAD_INVALID (416)

Open
#790 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Swift
Stars
8.9k
Forks
359
Avg merge
2d 20h
Merged PRs (30d)
13

Description

### Summary

`RegistryClient` blob push to **AWS ECR** can fail with HTTP **416** and error code **`BLOB_UPLOAD_INVALID`** when a large blob upload is interrupted or retried. ECR reports that the client restarted from byte **0** while the upload session already had bytes committed.

This is the underlying implementation issue tracked upstream in the **`container`** CLI repo: **apple/container#1895**

---

### Steps to reproduce

Reproducible via `container image push` (which uses this library). Minimal flow:

```sh
aws ecr get-login-password --region \
| container registry login --username AWS --password-stdin \
.dkr.ecr..amazonaws.com

container build \
--platform linux/amd64,linux/arm64 \
-t .dkr.ecr..amazonaws.com/: \
.

container image push .dkr.ecr..amazonaws.com/:
```

Most likely with **large multi-arch images** when a blob upload is partially sent then retried.

---

### Current behavior

Example error:

```
416 Range Not Satisfiable
{"errors":[{"code":"BLOB_UPLOAD_INVALID","message":"First byte of the layer part is 0 instead of 11304960"}]}
```

---

### Suspected root cause

**`Sources/ContainerizationOCI/Client/RegistryClient+Push.swift`**

Blob push flow:

1. `POST /v2//blobs/uploads/` → obtain upload session URL
2. Monolithic `PUT` of entire blob to `.../blobs/uploads/?digest=sha256:...`

The code passes a `bodyClosure` that resets the stream on retry:

```swift
// We have to pass a body closure rather than a body to reset the stream when retrying.
let bodyClosure = {
let stream = try streamGenerator()
return HTTPClientRequest.Body.stream(stream, length: .known(descriptor.size))
}
try await request(components: components, method: .PUT, bodyClosure: bodyClosure, ...)
```

**`Sources/ContainerizationOCI/Client/RegistryClient.swift`**

The generic `request()` loop retries by re-invoking `bodyClosure()` (stream from byte **0**) against the **same request URL**:

```swift
while true {
request.body = try bodyClosure()
let _response = try await client.execute(request, ...)
// retries on 5xx (shouldRetry) and network errors
continue
}
```

For blob uploads, if `PUT` partially succeeds then retries (network error or generic retry), the registry expects continuation from the last committed byte, not a restart from byte 0.

Similar issues in other OCI clients:
- [google/go-containerregistry#431](https://github.com/google/go-containerregistry/issues/431)
- [GoogleContainerTools/kaniko#607](https://github.com/GoogleContainerTools/kaniko/issues/607)

---

### Expected behavior

On blob upload failure or `BLOB_UPLOAD_INVALID`:

1. **Abandon** the stale upload session and issue a fresh `POST /blobs/uploads/`, then retry the full blob; **or**
2. **Resume** via distribution-spec chunked upload (`GET` upload status + `PATCH` with `Content-Range`).

Generic `request()` retry must not restart a blob `PUT` from byte 0 on the same upload session URL.

---

### Suggested fix

1. Refactor `push()` in `RegistryClient+Push.swift` to separate `initiateBlobUpload()` and `uploadBlob()`.
2. Wrap blob upload in an outer retry loop; on `BLOB_UPLOAD_INVALID` / HTTP 416, always start a **new upload session** (`POST`).
3. Disable or bypass generic `request()` retry for blob `PUT`, or pass `retryOptions: nil` for that call.
4. Add tests in `Tests/ContainerizationOCITests/` simulating partial upload + 416 → successful retry with new session.
5. (Optional) Implement `PATCH` + `Content-Range` resume per the [distribution spec](https://distribution.github.io/distribution/spec/api/#pushing-a-layer).

---

### Related

- **Upstream report (container CLI):** https://github.com/apple/container/issues/1895
- **Related container issues:** apple/container#1707 (ECR manifest PUT 401), apple/container#1001 (OCI index manifest registration)

Contributor guide

Open the contributing guide

Research direction

Read Sources/ContainerizationOCI/Client/RegistryClient+Push.swift and RegistryClient.swift to trace blob upload retries and body stream recreation. Then inspect Tests/ContainerizationOCITests/ for registry test patterns and add a partial-upload HTTP 416 case. Done means a failed blob upload abandons the stale session and retries through a new session without restarting on the same URL.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, swift
Domain
backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.