JanusGraph / JanusGraph/janusgraph

Ineffective storage strategy in ConsistentKeyIDAuthority

Open
#3,007 2 comments 1 reaction 0 assignees View on GitHub
kind/performance
Dominant language
Java
Stars
5.8k
Forks
1.2k
Avg merge
13h 53m
Merged PRs (30d)
6

Description

- Version: 0.6.0
- Storage Backend: All
- Mixed Index Backend: none

Current implementation of ConsistentKeyIDAuthority stores all allocated blocks of ids.
This lead to constant increase of data stored and eventually causes error in AerospikeStoreManager
(Aerospike allows to store only up to 1Mb per record)

To allocate new block of ids ConsistentKeyIDAuthority reads all this history data from storage but uses only the last id entry

[ConsistentKeyIDAuthority](https://github.com/JanusGraph/janusgraph/blob/0822cce3d7a50cdb530d13a363f076b18f9b14be/janusgraph-core/src/main/java/org/janusgraph/diskstorage/idmanagement/ConsistentKeyIDAuthority.java#L184)

My proposal is to keep only the last allocated rangeblock and remove elder ones

I've checked the code and think it should improve performance (no need to read and compare all range blocks) and shouldn't add any side effects

Contributor guide

Open the contributing guide

Research direction

Start with janusgraph-core/src/main/java/org/janusgraph/diskstorage/idmanagement/ConsistentKeyIDAuthority.java at the linked allocation logic. Read how allocated range blocks are stored and how AerospikeStoreManager's record limit affects that data. Done means only the latest allocated range block is retained, older blocks are removed, and future allocations no longer require reading the full history.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.