Support "very wide" clusters
- Dominant language
- Erlang
- Stars
- 7k
- Forks
- 1.1k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 9
Description
@janl:
> (probably a variant on cluster-aware clients) : Scaling out clustered-couch to "very-large" ops per second is an interesting experience.
>
>The basic premise that clusters contain all of the database is nice in that as you create copies of your data you get always-close behaviors, but there's a limit to individual cluster performance.
>
>Depending on the load-balancing involved in reaching your database, how spread your installation/clients are and much hot-spotting you experience, you can much more easily end up in cases where reads/writes have to be shunted to another cluster and because of replication-delay apps start to see strange behavior.
>
>One thing I've considered doing recently is using a transparent routing proxy on-top of couch that takes a provisioning configuration to locate clusters containing a specific database and use consistent-hashing to spread the keyspace transactions into predictable buckets, hinting back to the client what cluster-buckets were used.
>
>While not necessarily a couch-specific feature, and still subject to its own nuances, it's a useful lesson in scaling for the inevitable limit with 2.0 installations.
@wohali:
> Some of this may be possible with PSE and in-memory couch clones, or some sort of Redis compatibility layer. A lot of big big installs that I know have a Redis write-through cache that is hit first, that improves Couch performance drastically.
Contributor guide
Research direction
The issue discusses scaling CouchDB clusters through routing, consistent hashing, replication behavior, PSE, in-memory clones, and Redis compatibility. No files, tests, or entry points are named, and the thread does not define a concrete implementation or completion criteria; clarify the intended scope before starting.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- erlang, redis
- Domain
- databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100