JanusGraph / JanusGraph/janusgraph
Add ability to batch ElasticSearch updates when reindexing
- Dominant language
- Java
- Stars
- 5.8k
- Forks
- 1.2k
- Avg merge
- 13h 53m
- Merged PRs (30d)
- 6
Description
After upgrading to ES5 I found that re-indexing a relatively small graph got significantly slower. Traced it down to needing to set [index.translog.durability](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/index-modules-translog.html) to `async`. That speeds it up, but it does mean that there's some loss in reliability if there's a problem while reindexing. The ES recommendations on speeding up reindexing suggest [indexing several documents](https://www.elastic.co/guide/en/elasticsearch/reference/master/tune-for-indexing-speed.html) at once. Currently the ElasticSearch reindex code does use the `_bulk` endpoint, but it is sending a separate update index request for each vertex. Has there been discussion or plans to to cut down on the number of requests by using this method during reindexing?
Contributor guide
Research direction
Start by locating the Elasticsearch reindex code and tracing how it uses the _bulk endpoint and sends update requests for each vertex. The work is done when reindexing batches multiple document updates into fewer requests without changing the resulting index.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, java
- Domain
- databases, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100