google / google/go-containerregistry

crane: registry needs to hold full layers in memory

Open
#2,120 2 comments 0 reactions 0 assignees View on GitHub
bug lifecycle/stale
Dominant language
Go
Stars
4k
Forks
686
Avg merge
2d 12h
Merged PRs (30d)
26

Description

### Describe the bug

When using `crane registry serve` and pushing large layers with `PUT`, I observe excessive memory usage.
I traced this to the `PUT` handler function:
https://github.com/google/go-containerregistry/blob/59a4b85930392a30c39462519adc8a2026d47181/pkg/registry/blobs.go#L485

The in-flight blob is cached in full in a buffer until all chunks have been uploaded, and then `Put` is invoked after the transfer is complete.
This creates two problems:

- Excessive memory usage: with a typical CUDA base image, I can reliably produce OOM situations, since individual layers are gigabytes in size.
- The CLI side looks broken to the end-user: after every byte was sent via HTTP, the progress bar is at 100%, but the server still has to call `Put`. For a slow `BlobHandler` (one that has to upload data to a remote storage backend), this results in minutes of waiting where the progress bar is full and the user thinks that the upload may be stuck.

### To Reproduce

Terminal 1:

```
crane registry serve --address 127.0.0.1:1234
```

```
crane copy nvidia/cuda@sha256:f353ffca86e0cd93ab2470fe274ecf766519c24c37ed58cc2f91d915f7ebe53c localhost:1234/nvidia/cuda:latest
```

### Expected behavior

I want the memory usage to be reasonable, even for large blobs.

### Additional context

I'd like to provide a fix in the form of a custom `io.Reader` with an internal fixed-size buffer that allows a `BlobHandler` to read data in chunks.
This should make uploads faster, less memory intensive, and provide better feedback to users.
Is this an acceptable solution?

Contributor guide

Open the contributing guide

Research direction

Start with the PUT handler in pkg/registry/blobs.go around line 485 and inspect how BlobHandler receives completed uploads. Run crane registry serve with the provided crane copy reproduction, then verify large blob uploads no longer require the entire layer in memory and that the handler's post-upload delay is addressed.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.