developmentseed / developmentseed/multistore
Stream multipart UploadPart on Workers (avoid buffering parts in WASM memory)
- Dominant language
- Rust
- Stars
- 19
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
## Problem
Multipart `UploadPart` flows through the `NeedsBody` → `send_raw` path, which **buffers the entire part body in WASM linear memory** on the Cloudflare Workers runtime:
1. `collect_js_body` copies the request stream JS→WASM (`array_buffer()` + `to_vec()`)
2. `execute_multipart` computes SHA256 over the whole part for the outbound signature (`hash_payload`)
3. `send_raw` copies WASM→JS (`js_sys::Uint8Array::from`) to hand to `fetch`
So each part incurs **two full body copies across the JS/WASM boundary + an O(N) SHA256**, and is held in the worker's **128 MB memory** heap. Single-object `PutObject` does *not* have this problem — it uses the zero-copy `forward()` path (the `ReadableStream` is handed straight to `fetch`, signed with `UNSIGNED-PAYLOAD` so no hashing).
For typical part sizes (~8 MiB) this is a few ms of CPU and fine. But the 128 MB memory limit caps the maximum `multipart_chunksize` a client can use before the worker OOMs.
## Proposed fix
Route `UploadPart` through the streaming `forward()` path instead of `send_raw`:
- Generate a presigned URL for `PUT /{key}?partNumber=N&uploadId=X`. `object_store`'s `Signer::signed_url` only takes a `Path` (no query params), so presign manually via `S3RequestSigner` — it already builds SigV4 canonical query strings (incl. value-less params, fixed in #88).
- Return `HandlerAction::Forward` so the `ReadableStream` streams straight through `fetch`: no WASM buffering, no payload hash.
`CompleteMultipartUpload` stays on `send_raw` (its small XML body must be signed/forwarded), but that body is tiny. `CreateMultipartUpload`/`AbortMultipartUpload` are body-less and could also move to the forward path opportunistically.
## Interaction with Cloudflare's request-body size limit
Even with streaming, every request through a Worker is bounded by the [plan's request-body limit](https://developers.cloudflare.com/workers/platform/limits/#request-and-response-limits) (100 MB Free/Pro, 200 MB Business, 500 MB Enterprise default). Streaming changes which limit binds:
- **Today (buffered):** a part is capped by the *smaller* of the 128 MB memory limit and the plan body limit.
- **After streaming:** a part is capped only by the plan body limit (no WASM buffering).
So streaming lets clients use the full plan body limit for part sizes. It does **not** lift the body limit itself — that's a hard platform constraint (see separate doc note). Large objects must still be uploaded as multipart with part sizes ≤ the plan limit.
## Impact
- Removes the 128 MB part-size ceiling on Workers (raises it to the plan body limit).
- Eliminates 2 body copies + 1 SHA256 per part → lower CPU.
- Brings `UploadPart` to parity with the zero-copy `PutObject` path.
## References
- `crates/cf-workers/src/backend.rs` — `send_raw` (buffers); `forward` is the zero-copy template
- `crates/core/src/proxy.rs` — `execute_multipart`, `dispatch_operation` (the `NeedsBody` arm)
- Deferred from #88 to keep that PR scoped to correctness.
Contributor guide
Research direction
Read crates/cf-workers/src/backend.rs, comparing send_raw with the forward template, then trace execute_multipart and the NeedsBody arm in crates/core/src/proxy.rs. Follow S3RequestSigner for presigning query parameters; done means UploadPart streams through Forward without WASM body buffering or payload hashing while the other multipart operations retain their stated behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- api, backend
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100