Remote protocol: compress chunk transfer (zstd/gzip with Content-Encoding negotiation)
Nobody has claimed this yet.
- Dominant language
- C
- Stars
- 270
- Forks
- 20
- Avg merge
- 2h 26m
- Merged PRs (30d)
- 454
Description
Problem
Remote chunk transfer is uncompressed. Chunk payloads (prolly nodes, blobs) go over the wire raw, so push/clone of a large database moves far more bytes than necessary — especially over WAN links where bandwidth, not round trips, dominates.
Split out of #228 (secure remote protocol). The other items from that issue are done: TLS 1.3 + bearer-token auth, and batched chunk transfer (upload buffers into one POST /chunks; download pulls up to 256 chunks per POST /get-chunks). Compression is the remaining piece.
Current behavior
doltlite_http_remote.c/doltlite_remotesrv.c: request and response bodies are sent with noContent-Encoding; noAccept-Encodingis offered or honored.- The batched endpoints (
POST /chunksupload,POST /get-chunksdownload) send concatenated/framed raw chunk bytes.
Expected behavior
- Compress request/response bodies (zstd preferred, gzip acceptable) with content negotiation via
Accept-Encoding/Content-Encoding, so an older peer that doesn't advertise support still interoperates uncompressed. - Apply it to the bulk paths first: the framed
/get-chunksresponse and the/chunksupload body are where the bytes are.
Notes
- zstd would need to be vendored/linked (not currently a dependency); gzip via the already-linked zlib (
SQLITE_HAVE_ZLIB=1) is the lower-friction option if we want compression without a new dependency. - Negotiation must be backward compatible — the current wire format has no encoding header, so a compressing peer must fall back to raw when the other side doesn't advertise support.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read doltlite_http_remote.c and doltlite_remotesrv.c, then trace the framed POST /get-chunks response and POST /chunks upload body. Confirm how the current raw framing and request headers are handled before choosing the existing zlib path. Done means bulk bodies negotiate compression when supported and fall back to uncompressed transfer for older peers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, sqlite
- Domain
- databases, networking
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100