JanusGraph / JanusGraph/janusgraph

Add ability to batch ElasticSearch updates when reindexing

Open
#483 3 comments 4 reactions 0 assignees View on GitHub
index/elasticsearch
Dominant language
Java
Stars
5.8k
Forks
1.2k
Avg merge
13h 53m
Merged PRs (30d)
6

Description

After upgrading to ES5 I found that re-indexing a relatively small graph got significantly slower. Traced it down to needing to set [index.translog.durability](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/index-modules-translog.html) to `async`. That speeds it up, but it does mean that there's some loss in reliability if there's a problem while reindexing. The ES recommendations on speeding up reindexing suggest [indexing several documents](https://www.elastic.co/guide/en/elasticsearch/reference/master/tune-for-indexing-speed.html) at once. Currently the ElasticSearch reindex code does use the `_bulk` endpoint, but it is sending a separate update index request for each vertex. Has there been discussion or plans to to cut down on the number of requests by using this method during reindexing?

Contributor guide

Open the contributing guide

Research direction

Start by locating the Elasticsearch reindex code and tracing how it uses the _bulk endpoint and sends update requests for each vertex. The work is done when reindexing batches multiple document updates into fewer requests without changing the resulting index.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, java
Domain
databases, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.