apache / apache/couchdb

Mobile First replication protocol

Open
#1,503 0 comments 5 reactions 0 assignees View on GitHub
performance replication roadmap
Dominant language
Erlang
Stars
7k
Forks
1.1k
Avg merge
1d 16h
Merged PRs (30d)
9

Description

@janl:
>* needs HTTP2 / single connection
>* multipart/mixed with decompressed attachments (for browsers)
>The original protocol design didn’t take mobile devices into account and through PouchDB et.al. we are now learning that there are number of downsides to our protocol. We’ve helped a lot with introducing `_bulk_get`/`_revs`, but that’s more a bandaid than a considered strategy ;)
>
>That new version could also be HTTP/2-only, to take advantage of the new connection semantics there.

@daleharvey:
>So yup I think there are huge and not too complicated improvements here, its tricky as the improvements lie somewhere between the replicator and core api, but I think the 2 main ones off the top of my head are
>
>1. Mobile clients almost never need old deletions, an easy way to tell the replicator to ignore deletions that happen before you start replicating would be great (I think this could be fairly easily implemented by the replicator)
>
>2. The replication protocol is chatty, its optimised mostly for a case where there is a chance you have multiple filtered replications running, the actual data from the changes feed is never actually read it is just used to pick up revisions which are then read (the bulk_get). I believe it should be possible to have replication that is a stream of the full documents where changes have occured (deduped). This would basically be a formalised version of @nolanlawson's https://github.com/nolanlawson/pouchdb-replication-stream that took into account live replication (so basically gave users the advantages of replication-stream without needed to do any work, I believe this was in the order of 1000X speedups for large data sets)
>
>I think this may be tricky and we definitely need to consider edge cases (make sure we arent too bad with multiple filtered replications etc) but could be a monumentally large speedup

Contributor guide

Open the contributing guide

Research direction

Start by studying the replicator and core API areas involved in _bulk_get, _revs, the changes feed, and bulk_get. Compare the proposed single-connection HTTP/2 and full-document streaming approaches, including filtered and live replication edge cases. Done means a considered mobile-first replication protocol design with its tradeoffs and behavior defined.

Written by the indexing model from the issue text.

Assessment

Tech stack
erlang
Domain
backend-api-design, databases, distributed-systems, networking
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.