developmentseed / developmentseed/multistore

Stream multipart UploadPart on Workers (avoid buffering parts in WASM memory)

Open
#89 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
19
Forks
1
PR merge metrics
No merged PRs in 30d

Description

## Problem

Multipart `UploadPart` flows through the `NeedsBody` → `send_raw` path, which **buffers the entire part body in WASM linear memory** on the Cloudflare Workers runtime:

1. `collect_js_body` copies the request stream JS→WASM (`array_buffer()` + `to_vec()`)
2. `execute_multipart` computes SHA256 over the whole part for the outbound signature (`hash_payload`)
3. `send_raw` copies WASM→JS (`js_sys::Uint8Array::from`) to hand to `fetch`

So each part incurs **two full body copies across the JS/WASM boundary + an O(N) SHA256**, and is held in the worker's **128 MB memory** heap. Single-object `PutObject` does *not* have this problem — it uses the zero-copy `forward()` path (the `ReadableStream` is handed straight to `fetch`, signed with `UNSIGNED-PAYLOAD` so no hashing).

For typical part sizes (~8 MiB) this is a few ms of CPU and fine. But the 128 MB memory limit caps the maximum `multipart_chunksize` a client can use before the worker OOMs.

## Proposed fix

Route `UploadPart` through the streaming `forward()` path instead of `send_raw`:

- Generate a presigned URL for `PUT /{key}?partNumber=N&uploadId=X`. `object_store`'s `Signer::signed_url` only takes a `Path` (no query params), so presign manually via `S3RequestSigner` — it already builds SigV4 canonical query strings (incl. value-less params, fixed in #88).
- Return `HandlerAction::Forward` so the `ReadableStream` streams straight through `fetch`: no WASM buffering, no payload hash.

`CompleteMultipartUpload` stays on `send_raw` (its small XML body must be signed/forwarded), but that body is tiny. `CreateMultipartUpload`/`AbortMultipartUpload` are body-less and could also move to the forward path opportunistically.

## Interaction with Cloudflare's request-body size limit

Even with streaming, every request through a Worker is bounded by the [plan's request-body limit](https://developers.cloudflare.com/workers/platform/limits/#request-and-response-limits) (100 MB Free/Pro, 200 MB Business, 500 MB Enterprise default). Streaming changes which limit binds:

- **Today (buffered):** a part is capped by the *smaller* of the 128 MB memory limit and the plan body limit.
- **After streaming:** a part is capped only by the plan body limit (no WASM buffering).

So streaming lets clients use the full plan body limit for part sizes. It does **not** lift the body limit itself — that's a hard platform constraint (see separate doc note). Large objects must still be uploaded as multipart with part sizes ≤ the plan limit.

## Impact

- Removes the 128 MB part-size ceiling on Workers (raises it to the plan body limit).
- Eliminates 2 body copies + 1 SHA256 per part → lower CPU.
- Brings `UploadPart` to parity with the zero-copy `PutObject` path.

## References

- `crates/cf-workers/src/backend.rs` — `send_raw` (buffers); `forward` is the zero-copy template
- `crates/core/src/proxy.rs` — `execute_multipart`, `dispatch_operation` (the `NeedsBody` arm)
- Deferred from #88 to keep that PR scoped to correctness.

Contributor guide

Open the contributing guide

Research direction

Read crates/cf-workers/src/backend.rs, comparing send_raw with the forward template, then trace execute_multipart and the NeedsBody arm in crates/core/src/proxy.rs. Follow S3RequestSigner for presigning query parameters; done means UploadPart streams through Forward without WASM body buffering or payload hashing while the other multipart operations retain their stated behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
api, backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.