apache / apache/couchdb

Add support for collections

Open
#4,044 1 comment 0 reactions 0 assignees View on GitHub
enhancement needs-triage
Dominant language
Erlang
Stars
7k
Forks
1.1k
Avg merge
1d 16h
Merged PRs (30d)
9

Description

[NOTE]: # ( ^^ Provide a general summary of the request in the title above. ^^ )

## Summary

Map/reduce indexes may take a lot of time to be built on very large databases. In addition to working on improving the Query Server protocol (which is the topic of another ticket), another way to mitigate this problem would be to reduce the number of documents that a view needs to process. At least for me, each of my views begin with an "if" statement to check the document type, so it would probably make sense to give couchdb a way to know that only specific documents need to be passed to a certain view.

## Desired Behaviour

[NOTE]: # ( Tell us how the new feature should work. Be specific. )
[TIP]: # ( Do NOT give us access or passwords to your actual CouchDB! )

Exactly the same API than the one offered by the _partitions feature.

## Possible Solution

[NOTE]: # ( Not required. Suggest how to implement the addition or change. )
I believe we can take inspiration from the implementation of partitions. In particular the fact that partitions can have their own design documents only applying to the specific partition is exactly what is needed here. This is exactly the spirit of this proposal. Unfortunately there is a major caveat making partitions unsuitable for this use case, as per the documentation:

> A good partition has two basic properties. First, it should have a high cardinality. That is, a large partitioned database should have many more partitions than documents in any single partition. A database that has a single partition would be an anti-pattern for this feature. Secondly, the amount of data per partition should be “small”. The general recommendation is to limit individual partitions to less than ten gigabytes (10 GB) of data. Which, for the example of sensor documents, equates to roughly 60,000 years of data.

In the use case I describe, there would instead be a small number of partitions each having a large number of documents. This would therefore be the main difference between partitions and collections. Partitions are meant to hold the data of a specific sensor, while collections would be meant to store all documents being of type "sensor". This would still be a major improvement over passing the data of all the documents of the database to the view server to build an index since document serialization is a major cost center of the indexing process.

### Partitioned Collections

An interesting nice-to-have feature would be the ability to partition collections. And maybe conveniences making it easy for users to start with a non-partitioned collection, and then transition to a partitioned collection where relevant. For this purpose, maybe a collection could contain an optional "partition_key_properties" configuration option that can only be defined at collection creation time, and accepting either a simple value, or a composite key. A document not containing the properties would not be allowed to be added to the partitioned collection. A compliant document would automatically be added to a partition of the collection named after the document partition-key.

From that point, enabling users to replicate documents from one collection to another one of the same database would make it easy to transition from a non-partitioned collection to a partitioned collection. This is important since it can be very difficult to anticipate if partitions should be used or not, in practice the decision will often only be made after the need manifests itself in production.

### Migrating existing databases

Finally, to ease the transition for existing databases (extremely important), it should be possible to add an option to filtered replication allowing users to indicate that filtered documents should be added to a specific collection of the destination database.

Alternatively, similarly to the "partition_key_properties", every collection may have a "collection_key_properties" so that new documents added to the database and matching the rule are automatically added to the collection. In addition to this, for existing documents, such an option would require a full scan of the database every time collection_key_properties is modified. There is also the question of what to do if a document matches the collection_key_properties of multiple documents. Also, this would need to only affect documents not currently belonging to any collection.

Finally, an endpoint (potentially the all_docs endpoint) should still allow users to replicate all documents without replicating collections. This is to allow users to start from scratcg and reorganize the collection schema of their database (by first replicating all documents to a database no having any collection, and then replicating this database to the database with the new collection schema)

### Retaining the ability to define global views

Just like the partition features does not prevent users from defining global views, being able to define global views would also be of the essence for databases having collections. Now, maybe the collections features would also make it easy to add a "collections" property to global views enabling users to specify that only the documents of the listed collections must be sent to the view.

## Additional context

[TIP]: # ( Why does this feature matter to you? What unique circumstances do you have? )

Contributor guide

Open the contributing guide

Research direction

Start by reading the existing partitions implementation and API, then compare it with the Query Server, filtered replication, _all_docs, and global-view behavior described here. The proposal still needs a decided collection model and migration semantics before implementation can begin; done would include an agreed API and support for the required collection and view workflows.

Written by the indexing model from the issue text.

Assessment

Tech stack
erlang
Domain
api, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.