JanusGraph / JanusGraph/janusgraph
Ineffective storage strategy in ConsistentKeyIDAuthority
- Dominant language
- Java
- Stars
- 5.8k
- Forks
- 1.2k
- Avg merge
- 13h 53m
- Merged PRs (30d)
- 6
Description
- Version: 0.6.0
- Storage Backend: All
- Mixed Index Backend: none
Current implementation of ConsistentKeyIDAuthority stores all allocated blocks of ids.
This lead to constant increase of data stored and eventually causes error in AerospikeStoreManager
(Aerospike allows to store only up to 1Mb per record)
To allocate new block of ids ConsistentKeyIDAuthority reads all this history data from storage but uses only the last id entry
[ConsistentKeyIDAuthority](https://github.com/JanusGraph/janusgraph/blob/0822cce3d7a50cdb530d13a363f076b18f9b14be/janusgraph-core/src/main/java/org/janusgraph/diskstorage/idmanagement/ConsistentKeyIDAuthority.java#L184)
My proposal is to keep only the last allocated rangeblock and remove elder ones
I've checked the code and think it should improve performance (no need to read and compare all range blocks) and shouldn't add any side effects
Contributor guide
Research direction
Start with janusgraph-core/src/main/java/org/janusgraph/diskstorage/idmanagement/ConsistentKeyIDAuthority.java at the linked allocation logic. Read how allocated range blocks are stored and how AerospikeStoreManager's record limit affects that data. Done means only the latest allocated range block is retained, older blocks are removed, and future allocations no longer require reading the full history.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, distributed-systems
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100