aws / aws/containers-roadmap

[ECR] [bug]: OCI chunked blob upload (PATCH) silently discards data for chunks after the first

Open
#2,831 3 comments 7 reactions 0 assignees View on GitHub
ECR
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

ECR's OCI chunked blob upload silently discards the body of every PATCH request after the first. Each subsequent chunk is accepted with HTTP 201 but the `Range` response header does not advance, making it impossible to upload blobs that require more than one intermediate PATCH.

## Expected behaviour (OCI Distribution Spec)

Per the [OCI Distribution Spec §Pushing Blobs](https://github.com/opencontainers/distribution-spec/blob/main/spec.md):

- Each intermediate PATCH **MUST** return `202 Accepted`
- `Content-Range: {first}-{last}` indicates the position of the chunk within the **full blob**; the request body contains those bytes starting at offset 0
- The `Range` response header **MUST** reflect the total bytes received so far and **MUST** advance after each successful PATCH

## Actual behaviour

- Intermediate PATCH requests return `201 Created` instead of `202 Accepted`
- The `Range` response header advances correctly after the **first** PATCH
- The `Range` response header does **not** advance after the second or any subsequent PATCH — the body is silently discarded
- Omitting `Content-Range` entirely on the second PATCH produces the same result (body discarded, Range unchanged)

ECR seems to read `body[Content-Range.start .. Content-Range.end]` from the request body instead of reading the body from offset 0. For the first PATCH `Content-Range.start = 0`, so ECR reads from the beginning of the body — which is coincidentally correct. For any subsequent PATCH `Content-Range.start > 0`, ECR seeks past the end of the body (which is only `end - start + 1` bytes long), reads nothing, and silently accepts the request without storing any data.

This was confirmed by sending a padded body on the second PATCH:

```
Content-Range: 10485760-20971519
Body: 10 MB of zeros + 10 MB of real chunk data (20 MB total)
```

ECR reads `body[10485760 .. 20971519]` = the real chunk data, and `Range` advances to `0-20971519`. The `Range` only advances when the body contains the actual bytes at the absolute offset matching `Content-Range.start`.

## Reproduction

```bash
REGISTRY=".dkr.ecr..amazonaws.com"
REPO=""
AUTH="" # from docker login

# Create test data: 10 MB chunk of 0x01, 10 MB chunk of 0x02
python3 -c "
open('chunk1.bin','wb').write(bytes([0x01])*10485760)
open('chunk2.bin','wb').write(bytes([0x02])*10485760)
"

# POST — initiate upload
RESP=$(curl -si -X POST \
-H "Authorization: Basic $AUTH" \
"https://$REGISTRY/v2/$REPO/blobs/uploads/")
echo "$RESP" | grep -E "^HTTP|^[Ll]ocation:|^[Rr]ange:|^OCI"
LOCATION=$(echo "$RESP" | grep -i "^location:" | tr -d '\r' | awk '{print $2}')

# PATCH 1 — Content-Range: 0-10485759, body = chunk1 (0x01)
# Range advances: 0-10485759 ✓
RESP=$(curl -si -X PATCH \
-H "Authorization: Basic $AUTH" \
-H "Content-Type: application/octet-stream" \
-H "Content-Range: 0-10485759" \
-H "Content-Length: 10485760" \
--data-binary @chunk1.bin \
"$LOCATION")
echo "$RESP" | grep -E "^HTTP|^[Rr]ange:"
LOCATION=$(echo "$RESP" | grep -i "^location:" | tr -d '\r' | awk '{print $2}')

# PATCH 2 — Content-Range: 10485760-20971519, body = chunk2 (0x02)
# Range does NOT advance: still 0-10485759 ✗
RESP=$(curl -si -X PATCH \
-H "Authorization: Basic $AUTH" \
-H "Content-Type: application/octet-stream" \
-H "Content-Range: 10485760-20971519" \
-H "Content-Length: 10485760" \
--data-binary @chunk2.bin \
"$LOCATION")
echo "$RESP" | grep -E "^HTTP|^[Rr]ange:"
LOCATION=$(echo "$RESP" | grep -i "^location:" | tr -d '\r' | awk '{print $2}')
```

Expected output after PATCH 2: `Range: 0-20971519`
Actual output after PATCH 2: `Range: 0-10485759`

Omitting Content-Range entirely on PATCH 2 produces the same result.

## Impact

Any OCI client that performs chunked blob upload for blobs larger than `OCI-Chunk-Min-Length` (10485760, returned in the POST response) will silently fail — only the first chunk is stored. The upload appears to succeed (PUT returns `201` with a valid `Docker-Content-Digest`) but the stored blob is truncated to the first chunk. Docker CLI and crane are unaffected because they use single-request upload (one PATCH or one PUT for the entire blob), which never exercises the second-chunk path.

## Environment

- Region: us-east-1 (also reproduced on us-west-2)
- ECR API version: registry/2.0 (from `Docker-Distribution-Api-Version` header)
- Tested: June 2026

Contributor guide

Open the contributing guide

Research direction

Start with the supplied curl reproduction and compare the PATCH behavior with OCI Distribution Spec §Pushing Blobs. Verify that a second chunk is stored, the intermediate response is 202, and the Range header advances through the complete upload; the payload names no repository files or tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
cloud
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.