dolthub / dolthub/doltlite

Remote protocol: compress chunk transfer (zstd/gzip with Content-Encoding negotiation)

Open
#1,584 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
270
Forks
20
Avg merge
2h 26m
Merged PRs (30d)
454

Description

Problem

Remote chunk transfer is uncompressed. Chunk payloads (prolly nodes, blobs) go over the wire raw, so push/clone of a large database moves far more bytes than necessary — especially over WAN links where bandwidth, not round trips, dominates.

Split out of #228 (secure remote protocol). The other items from that issue are done: TLS 1.3 + bearer-token auth, and batched chunk transfer (upload buffers into one POST /chunks; download pulls up to 256 chunks per POST /get-chunks). Compression is the remaining piece.

Current behavior

  • doltlite_http_remote.c / doltlite_remotesrv.c: request and response bodies are sent with no Content-Encoding; no Accept-Encoding is offered or honored.
  • The batched endpoints (POST /chunks upload, POST /get-chunks download) send concatenated/framed raw chunk bytes.

Expected behavior

  • Compress request/response bodies (zstd preferred, gzip acceptable) with content negotiation via Accept-Encoding / Content-Encoding, so an older peer that doesn't advertise support still interoperates uncompressed.
  • Apply it to the bulk paths first: the framed /get-chunks response and the /chunks upload body are where the bytes are.

Notes

  • zstd would need to be vendored/linked (not currently a dependency); gzip via the already-linked zlib (SQLITE_HAVE_ZLIB=1) is the lower-friction option if we want compression without a new dependency.
  • Negotiation must be backward compatible — the current wire format has no encoding header, so a compressing peer must fall back to raw when the other side doesn't advertise support.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read doltlite_http_remote.c and doltlite_remotesrv.c, then trace the framed POST /get-chunks response and POST /chunks upload body. Confirm how the current raw framing and request headers are handled before choosing the existing zlib path. Done means bulk bodies negotiate compression when supported and fall back to uncompressed transfer for older peers.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, sqlite
Domain
databases, networking
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.