CentreForDigitalHumanities / CentreForDigitalHumanities/Textcavator
Improve strategy for elasticsearch shard count
- Dominant language
- Python
- Stars
- 12
- Forks
- 3
- Avg merge
- 3d 3m
- Merged PRs (30d)
- 9
Description
For context on this, see: https://www.elastic.co/docs/deploy-manage/production-guidance/optimize-performance/size-shards
TLDR: for optimal searching, the shard count for an Elasticsearch index should depend on the amount of documents and the size of the data.
**Is your feature request related to a problem? Please describe.**
Our current strategy to determine the shard count for an Elasticsearch index is:
- In development settings, use 1 shard.
- In production settings, use 5 shards.
I think the idea behind this strategy is that you probably use small sample datasets in development and large datasets in production. But this has some issues:
- Tying the shard count to development/production is not great; sometimes you do use large datasets in development.
- Our corpora vary in size by several orders of magnitude. 5 shards is often too many (e.g. for the Troonredes corpus) or too few (e.g. for the Delpher corpus).
In theory, you can customise the shard count the `es_settings` in the corpus definition. (In development, you can now also override the `es_settings` through Django settings to set fewer shards.) In practice, we never do this. Also, we have no strategy determine what the shard count should be.
**Describe the solution you'd like**
I'm wondering if we could automatically estimate a good shard count before indexing.
We don't (always) know the number of documents up front, but if the source data is stored on disk, you could use that to estimate the size of the data. Usually, the size of the indexed data is more or less equal to the size of the source files. (Not always, but it's better than nothing.)
So: size up the source data, then go with the recommendation that shards should be between 10 and 50 GB, and you should get a reasonable shard count.
**Describe alternatives you've considered**
We could put in more effort to adjust the shard count manually. However:
- We don't have a straightforward way to determine the shard count.
- Hard-coding the shard count on the corpus is an issue when you work with sample data.
A more exact strategy would be to make an initial index, *then* determine the shard count, set up a new index and migrate the data. (You cannot change the shard count of an existing index.) This would be more accurate in estimating the size of the stored data, and could also take the number of documents into account. But it's obviously inefficient.
(Nonetheless, determining the shard count from an existing index could be useful in some cases, e.g. when you're updating corpus. If an index already exists, this method is faster and more accurate.)
Contributor guide
No contributing guide indexed for this repository
Research direction
The issue names no files or tests. Start by locating the current development/production shard strategy, corpus `es_settings`, and the indexing entry point; then trace how source data and existing indices are available. Done should mean a documented, tested strategy that estimates an appropriate shard count from available corpus data without relying only on the environment.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch
- Domain
- databases, search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100