apache / apache/couchdb

Support "very wide" clusters

Open
#1,527 4 comments 0 reactions 0 assignees View on GitHub
cluster performance roadmap
Dominant language
Erlang
Stars
7k
Forks
1.1k
Avg merge
1d 16h
Merged PRs (30d)
9

Description

@janl:
> (probably a variant on cluster-aware clients) : Scaling out clustered-couch to "very-large" ops per second is an interesting experience.
>
>The basic premise that clusters contain all of the database is nice in that as you create copies of your data you get always-close behaviors, but there's a limit to individual cluster performance.
>
>Depending on the load-balancing involved in reaching your database, how spread your installation/clients are and much hot-spotting you experience, you can much more easily end up in cases where reads/writes have to be shunted to another cluster and because of replication-delay apps start to see strange behavior.
>
>One thing I've considered doing recently is using a transparent routing proxy on-top of couch that takes a provisioning configuration to locate clusters containing a specific database and use consistent-hashing to spread the keyspace transactions into predictable buckets, hinting back to the client what cluster-buckets were used.
>
>While not necessarily a couch-specific feature, and still subject to its own nuances, it's a useful lesson in scaling for the inevitable limit with 2.0 installations.

@wohali:
> Some of this may be possible with PSE and in-memory couch clones, or some sort of Redis compatibility layer. A lot of big big installs that I know have a Redis write-through cache that is hit first, that improves Couch performance drastically.

Contributor guide

Open the contributing guide

Research direction

The issue discusses scaling CouchDB clusters through routing, consistent hashing, replication behavior, PSE, in-memory clones, and Redis compatibility. No files, tests, or entry points are named, and the thread does not define a concrete implementation or completion criteria; clarify the intended scope before starting.

Written by the indexing model from the issue text.

Assessment

Tech stack
erlang, redis
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.