cloudflare / cloudflare/serverless-registry

Performance Bottleneck and OOM in PATCH Requests During Large Image Pushes

Open
#119 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
1.5k
Forks
122
PR merge metrics
No merged PRs in 30d

Description

## Description
Currently, the Registry Worker faces significant stability and performance issues when handling large image pushes, specifically during the `PATCH` phase of chunked uploads.

## Observed Problems
1. **Memory Exhaustion (OOM):** In `src/router.ts`, when a `PATCH` request arrives without a `Content-Length` header, the code executes `await req.blob()`. This action buffers the entire chunk into the Worker's memory. For large chunks (e.g., 50MB-100MB+), this frequently exceeds the Worker's memory limits, causing immediate crashes.
2. **Request Timeouts:** Buffering the entire request body before initiating the upload to R2 consumes a significant portion of the request's time limit. This leads to 10-minute timeouts on slower connections or with very large chunks.
3. **Broken UX (Progress Jitter):** From the user's perspective, the CLI progress bar (e.g., in `docker push` or `regctl`) freezes while the Worker is buffering, and then "jumps" forward once the R2 upload finally begins. This makes the transfer appear unreliable or "stuck".
4. **Protocol Parsing Fragility:** The current parsing of the `Content-Range` header is brittle and fails to handle common variations (like the `bytes ` prefix), leading to `416 Range Not Satisfiable` errors even when the client is behaving correctly.

## Impact
- Inability to push large layers (typically > 100MB).
- High error rates in production due to OOM.
- Poor user experience due to perceived hangs and sudden jumps in transfer progress.

## Test
I'm using `regctl`, which is friendly to chunked uploads.
```bash
# regctl registry config
{
"hosts": {
"": {
"tls": "enabled",
"hostname": ""
},
"": {
"tls": "enabled",
"regcert": "system",
"hostname": "",
"api": "default",
"blobChunk": 52428800,
"blobMax": 52428800,
"reqPerSec": 10,
"reqConcurrent": 4
}
}
}
```

```bash
# regctl image copy --fast /xxxxxxxx/xxxx/xxxxxx:1.0.1 /xxxxxxxx/xxxx/xxxxxx:1.0.1
time=2026-02-05T16:52:27.711+08:00 level=WARN msg="API field has been deprecated" api=default host=
time=2026-02-05T16:52:27.711+08:00 level=WARN msg="Changing reqPerSec settings for registry" orig=3 new=4 host=
time=2026-02-05T16:52:27.730+08:00 level=WARN msg="failed to setup CA pool" err="failed to load host specific ca (registry: ): pem.Decode is nil: system"
sha256:da3b410 [========================================] 100.00% 1.143kB/1.143kB
sha256:1c8e5e0 [========================================] 100.00% 60.062MB/60.062MB
sha256:c415b60 [==> ] 6.86% 52.429MB/764.153MB
Manifests: 2/5 | Blobs: 250.613MB copied, 883.550MB skipped, 711.724MB queued | Elapsed: 784s
```
It keeps getting stuck at the chunk threshold of 52MB.

Contributor guide

Open the contributing guide

Research direction

Start by reading the PATCH upload path in src/router.ts, focusing on handling requests without Content-Length and parsing Content-Range. Reproduce the issue with the provided regctl configuration and large image copy command; done means large chunks avoid worker OOMs and timeouts, progress no longer stalls, and valid bytes-prefixed ranges are accepted.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, typescript
Domain
api, backend, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.