JanusGraph / JanusGraph/janusgraph

import speed is very slow

Open
#2,041 1 comment 0 reactions 0 assignees View on GitHub
kind/bug/possible
Dominant language
Java
Stars
5.8k
Forks
1.2k
Avg merge
13h 53m
Merged PRs (30d)
6

Description

Please include configurations and logs if available.
```properties
gremlin.graph= org.janusgraph.core.JanusGraphFactory
storage.backend= cql
storage.cql.keyspace=graphtest
query.batch= true
query.force-index= true
storage.cql.compression= true
storage.cql.compression-block-size=4
storage.cql.read-consistency-level=LOCAL_ONE
storage.cql.write-consistency-level=LOCAL_QUORUM
storage.cql.local-core-connections-per-host=2
storage.cql.local-max-connections-per-host= 2
storage.cql.replication-factor=1
storage.hostname=9.109.52.166
ids.block-size= 10000000
schema.default=none
storage.batch-loading=true
storage.buffer-size=2048
index.ES.backend= elasticsearch
index.ES.hostname=9.109.52.166
index.ES.elasticsearch.client-only= true
index.ES.index-name= graphtest
index.ES.elasticsearch.create.ext.index.number_of_shards= 4
index.ES.elasticsearch.create.ext.index.number_of_replicas= 0
index.ES.elasticsearch.create.ext.index.refresh_interval= 600s
log.tx.key-consistent=true
cache.db-cache-size= 0.0
cache.db-cache-time= 0
graph.replace-instance-if-exists=true
```
when id use janus package to import data properties is above code is below .
the speed is very slow in single thread. 3M/min when data is little.
when data comes large, the data import is very slow.

For confirmed bugs, please report:
- Version: 0.3.1
- Storage Backend: cql
- Mixed Index Backend: es
- Mailing list Thread URL:
- Steps to Reproduce:

### my data import code

```java
startNs = System.nanoTime();
JanusGraph graph = open(configurationFile, graphName);
GraphTraversalSource g = graph.traversal();
janusTime += System.nanoTime() -startNs;
reader = new BufferedReader(new InputStreamReader(fs.open(path)));
String firstLine = reader.readLine();
HashMap rowMap = parseRowMap(firstLine, elementType, labelName);

if(rowMap == null){
logger.error(" parse file header of file {} failed, pass this file", fileName);
return;
}

String line;
Long count = new Long(0);
if (elementType.equals(vertexPrefix)) {
HashMap elementMap = new HashMap<>();
while ((line = reader.readLine()) != null) {
elementMap.clear();
String[] lineValues = line.split(dataSeperator);
if(lineValues.length != rowMap.size()){
logger.warn("pass invalid data {} th line of file {}:, do not import this line", count, fileName);
count += 1;
continue;
}
// read data
for(int i = 0; i < lineValues.length; i++){
Integer index = Integer.valueOf(i);
elementMap.put(rowMap.get(index), lineValues[i]);
}
elementMap.put(typeForKgSchema, labelName);
// add property by g.addV().property()
importOneVertex(elementMap, g, labelName, count);
count += 1;
if((count + 1) % batchSize == 0){
Long startNs1 = System.nanoTime();
g.tx().commit();
commitTime += System.nanoTime() - startNs1;
}
if (count % 10000 == 0){
logger.info("processed {} lines " + elementType, count);
}
}
Long startNs1 = System.nanoTime();
g.tx().commit();
commitTime += System.nanoTime() - startNs1;
}
```

Contributor guide

Open the contributing guide

Research direction

Start with the supplied Java importer, especially importOneVertex and the g.tx().commit() calls, and reproduce the slowdown using the listed Cassandra and Elasticsearch configuration. Profile the import as data volume grows and compare parsing, vertex creation, and commit time; done means the bottleneck is identified and the import-speed behavior is corrected or clearly documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
cassandra, elasticsearch, java
Domain
databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.