JanusGraph / JanusGraph/janusgraph
Empty CachedVertex pollute query results of embeded lucene mixed index
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 5.8k
- Forks
- 1.2k
- Avg merge
- 13h 53m
- Merged PRs (30d)
- 6
Description
Configuration
```
# storage backend
gremlin.graph=org.janusgraph.core.JanusGraphFactory
storage.backend=cassandrathrift
storage.hostname=192.168.99.100
schema.default=none
# index backend
# path is relative to pwd
index.search.directory=build/searchindex
```
When trying to use search mixedIndex with embeded lucene:
```
SearchResults results = new SearchResults<>(query);
String queryString = getQueryString(query.getSearchString(), getPropsForSearchIndexes());
// run query against index
JanusGraphIndexQuery indexQuery = janusGraph.indexQuery(INDEX_NAME, queryString);
if (query.isLimited()) {
indexQuery.limit(query.getLimit());
}
indexQuery.offset(query.getOffset());
int vertexCount = 0;
for (Result result : indexQuery.vertices()) {
Vertex vertex = result.getElement();
vertexCount++;
// When using built-in lucene index backend we get CachedVertex objects which do not have properties set.
// That is why we need to check if vertex was created by DataService (has already assigned ID to it).
// This also means that paging is unreliable and should not be used with embedded lucene.
if (vertex.property(Storage.ID).isPresent()) {
results.addResult(new GraphResult(result.getElement().property(Storage.ID).value(), result.getScore()));
}
}
if (query.isPaged()) {
// no janus interface to get total count, so re-querying is the only way
// TODO improve performance or allow to disable counting
Iterable> it = janusGraph.indexQuery(INDEX_NAME, queryString).vertices();
results.setTotalCount((int) StreamSupport.stream(it.spliterator(), false).count());
} else {
results.setTotalCount(results.size());
}
LOG.debug("Index query returned raw vertices: " + vertexCount);
LOG.debug("Index query returned matches: " + results.size());
return results;
}
```
We get:
```
2017-05-19 14:02:13,108 [main] graphdb.database.IndexSerializer INFO Converted query string with 2 replacements: [v."short_description":db* OR v."name":db*] => [short_description:db* OR name:db*]
2017-05-19 14:02:13,166 [main] janus.search.IndexQueryExecutor DEBUG Index query returned raw vertices: 8
2017-05-19 14:02:13,166 [main] janus.search.IndexQueryExecutor DEBUG Index query returned matches: 4
```
The issue is intermittent and does NOT always occur.
When it does occur the results list size is doubled and each 'good' result has a CachedVertex with no properties set.
We can work-around this by manually filtering result list as shown in above code, but this makes using limit and offset options unreliable.
This issues does not happen when using solr-cloud index backend confiugration.
```
# storage backend
gremlin.graph=org.janusgraph.core.JanusGraphFactory
storage.backend=cassandrathrift
storage.hostname=9.128.105.70
schema.default=none
# index backend
index.search.backend=solr
index.search.solr.mode=cloud
index.search.solr.zookeeper-url=9.128.105.70:2182
```
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the issue with the embedded Lucene configuration and the shown janusGraph.indexQuery(INDEX_NAME, queryString) path. Compare raw index results with CachedVertex properties, especially when limit and offset are used, and contrast the behavior with the Solr configuration. Done means index queries no longer add empty duplicate vertices and paging produces reliable results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cassandra, java
- Domain
- databases, search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100