JanusGraph / JanusGraph/janusgraph

improve design of elasticsearch id

Open
#1,303 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

index/elasticsearch
Dominant language
Java
Stars
5.8k
Forks
1.2k
Avg merge
13h 53m
Merged PRs (30d)
6

Description

I have ten billion records for mixed indexes , as load data into elasticsearch I find janusgraph use index mapping like

{
"_index" : "allindex",
"_type" : "big",
"_id" : "38w",
"_score" : 1.0,
"_source" : {
"domain_name" : "willam.boss.com"
}

_id is the document id of elasticearch , janusgraph create this id use LongEncoding.encode(vertexid)
this mean janusgraph assign document document id for elasticsearch ,this will decrease the speed of data loading of elasticsearch _bulk api
in source code of elasticearch

private void innerCreate(Create create) throws IOException {
if (engineConfig.isOptimizeAutoGenerateId() && create.autoGeneratedId() && !create.canHaveDuplicates()) {
// We don't need to lock because this ID cannot be concurrently updated:
innerCreateNoLock(create, Versions.NOT_FOUND, null);
} else {
synchronized (dirtyLock(create.uid())) {
final long currentVersion;
final VersionValue versionValue;
versionValue = versionMap.getUnderLock(create.uid().bytes());
if (versionValue == null) {
currentVersion = loadCurrentVersionFromIndex(create.uid());
} else {
if (engineConfig.isEnableGcDeletes() && versionValue.delete() && (engineConfig.getThreadPool().estimatedTimeInMillis() - versionValue.time()) > engineConfig.getGcDeletesInMillis()) {
currentVersion = Versions.NOT_FOUND; // deleted, and GC
} else {
currentVersion = versionValue.version();
}
}
innerCreateNoLock(create, currentVersion, versionValue);
}
}
}

this means if we don't use OptimizeAutoGenerateId elasticsearch will use check each record of index ,this will use up the read io of your computer ! when you have 100milliion the speed will slow quickly !
Hope you will improve this

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the issue's Elasticsearch innerCreate example and tracing where JanusGraph assigns IDs with LongEncoding.encode(vertexid). Identify the relevant ID-generation and bulk-loading entry points before proposing an approach. Done would require an agreed design and evidence that large bulk loads avoid the reported read-I/O cost.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, java
Domain
backend, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.