JanusGraph / JanusGraph/janusgraph

Empty CachedVertex pollute query results of embeded lucene mixed index

Open
#312 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

kind/bug/possible
Dominant language
Java
Stars
5.8k
Forks
1.2k
Avg merge
13h 53m
Merged PRs (30d)
6

Description

Configuration
```
# storage backend
gremlin.graph=org.janusgraph.core.JanusGraphFactory
storage.backend=cassandrathrift
storage.hostname=192.168.99.100
schema.default=none

# index backend
# path is relative to pwd
index.search.directory=build/searchindex
```

When trying to use search mixedIndex with embeded lucene:

```
SearchResults results = new SearchResults<>(query);
String queryString = getQueryString(query.getSearchString(), getPropsForSearchIndexes());

// run query against index
JanusGraphIndexQuery indexQuery = janusGraph.indexQuery(INDEX_NAME, queryString);
if (query.isLimited()) {
indexQuery.limit(query.getLimit());
}
indexQuery.offset(query.getOffset());

int vertexCount = 0;
for (Result result : indexQuery.vertices()) {
Vertex vertex = result.getElement();
vertexCount++;

// When using built-in lucene index backend we get CachedVertex objects which do not have properties set.
// That is why we need to check if vertex was created by DataService (has already assigned ID to it).
// This also means that paging is unreliable and should not be used with embedded lucene.
if (vertex.property(Storage.ID).isPresent()) {
results.addResult(new GraphResult(result.getElement().property(Storage.ID).value(), result.getScore()));
}
}

if (query.isPaged()) {
// no janus interface to get total count, so re-querying is the only way
// TODO improve performance or allow to disable counting
Iterable> it = janusGraph.indexQuery(INDEX_NAME, queryString).vertices();
results.setTotalCount((int) StreamSupport.stream(it.spliterator(), false).count());
} else {
results.setTotalCount(results.size());
}

LOG.debug("Index query returned raw vertices: " + vertexCount);
LOG.debug("Index query returned matches: " + results.size());

return results;
}
```

We get:
```
2017-05-19 14:02:13,108 [main] graphdb.database.IndexSerializer INFO Converted query string with 2 replacements: [v."short_description":db* OR v."name":db*] => [short_description:db* OR name:db*]
2017-05-19 14:02:13,166 [main] janus.search.IndexQueryExecutor DEBUG Index query returned raw vertices: 8
2017-05-19 14:02:13,166 [main] janus.search.IndexQueryExecutor DEBUG Index query returned matches: 4
```

The issue is intermittent and does NOT always occur.
When it does occur the results list size is doubled and each 'good' result has a CachedVertex with no properties set.

We can work-around this by manually filtering result list as shown in above code, but this makes using limit and offset options unreliable.

This issues does not happen when using solr-cloud index backend confiugration.
```
# storage backend
gremlin.graph=org.janusgraph.core.JanusGraphFactory
storage.backend=cassandrathrift
storage.hostname=9.128.105.70
schema.default=none

# index backend
index.search.backend=solr
index.search.solr.mode=cloud
index.search.solr.zookeeper-url=9.128.105.70:2182
```

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the issue with the embedded Lucene configuration and the shown janusGraph.indexQuery(INDEX_NAME, queryString) path. Compare raw index results with CachedVertex properties, especially when limit and offset are used, and contrast the behavior with the Solr configuration. Done means index queries no longer add empty duplicate vertices and paging produces reliable results.

Written by the indexing model from the issue text.

Assessment

Tech stack
cassandra, java
Domain
databases, search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.