JanusGraph / JanusGraph/janusgraph

JanusGraph SchemaAction.REINDEX does not remove stale data from index backend

Open
#354 2 comments 0 reactions 0 assignees View on GitHub
kind/bug/possible
Dominant language
Java
Stars
5.8k
Forks
1.2k
Avg merge
13h 53m
Merged PRs (30d)
6

Description

According to:
http://docs.janusgraph.org/0.1.0/indexes.html

When you do:
```
JanusGraphManagement mgmt = graph.openManagement();
JanusGraphIndex index = mgmt.getGraphIndex("indexName");
mgmt.updateIndex(index, SchemaAction.REINDEX).get();
mgmt.commit();
```

You'd expect that index data is fully representative of what is currently stored in the graph database.
Still the database scanning works only one way: all existing vertices from storage backend are re-added to index backend.

Sometimes when vertex deletion does not get propagated to the index (e.g see #329), index can contain vertices that are no longer in the graph.

In causes the following issues:
1) Queries that use the index will still return such deleted vertices with their ids without performing any checks or logging any errors
2) Reindexing action does not fix this issue

The only workaround is to drop/clear the index manually before re-indexing.
This is often time consuming and leaves index non-operational until re-indexing is completely finished.

Reindexing action could be improved to:
- incrementally query the index backend and check the existence of vertex ids in the graph
- remove stale data from the index

Quick and dirty way to re-create:
Create mixed index, fill-in some data, drop cassandra storage backend (or some part of if) to simulate failure, restart janus and run some queries using mixed index.

Contributor guide

Open the contributing guide

Research direction

Start at the JanusGraphManagement.updateIndex entry point and the SchemaAction.REINDEX path, then reproduce the stale-index case described with a mixed index and Cassandra storage. Trace how existing graph vertices are scanned and how the index backend is updated. Done means REINDEX also removes index entries for vertices no longer present, with coverage for the reported query behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
cassandra, java
Domain
databases, search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.