JanusGraph / JanusGraph/janusgraph
improve design of elasticsearch id
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 5.8k
- Forks
- 1.2k
- Avg merge
- 13h 53m
- Merged PRs (30d)
- 6
Description
I have ten billion records for mixed indexes , as load data into elasticsearch I find janusgraph use index mapping like
{
"_index" : "allindex",
"_type" : "big",
"_id" : "38w",
"_score" : 1.0,
"_source" : {
"domain_name" : "willam.boss.com"
}
_id is the document id of elasticearch , janusgraph create this id use LongEncoding.encode(vertexid)
this mean janusgraph assign document document id for elasticsearch ,this will decrease the speed of data loading of elasticsearch _bulk api
in source code of elasticearch
private void innerCreate(Create create) throws IOException {
if (engineConfig.isOptimizeAutoGenerateId() && create.autoGeneratedId() && !create.canHaveDuplicates()) {
// We don't need to lock because this ID cannot be concurrently updated:
innerCreateNoLock(create, Versions.NOT_FOUND, null);
} else {
synchronized (dirtyLock(create.uid())) {
final long currentVersion;
final VersionValue versionValue;
versionValue = versionMap.getUnderLock(create.uid().bytes());
if (versionValue == null) {
currentVersion = loadCurrentVersionFromIndex(create.uid());
} else {
if (engineConfig.isEnableGcDeletes() && versionValue.delete() && (engineConfig.getThreadPool().estimatedTimeInMillis() - versionValue.time()) > engineConfig.getGcDeletesInMillis()) {
currentVersion = Versions.NOT_FOUND; // deleted, and GC
} else {
currentVersion = versionValue.version();
}
}
innerCreateNoLock(create, currentVersion, versionValue);
}
}
}
this means if we don't use OptimizeAutoGenerateId elasticsearch will use check each record of index ,this will use up the read io of your computer ! when you have 100milliion the speed will slow quickly !
Hope you will improve this
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the issue's Elasticsearch innerCreate example and tracing where JanusGraph assigns IDs with LongEncoding.encode(vertexid). Identify the relevant ID-generation and bulk-loading entry points before proposing an approach. Done would require an agreed design and evidence that large bulk loads avoid the reported read-I/O cost.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, java
- Domain
- backend, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100