JanusGraph / JanusGraph/janusgraph
Support ASCII String, UUID, byte[] custom vertex ids
- Dominant language
- Java
- Stars
- 5.8k
- Forks
- 1.2k
- Avg merge
- 13h 53m
- Merged PRs (30d)
- 6
Description
This is an Epic issue to add multiple different possible data types as vertex ids into JanusGraph.
The issue regarding adding custom String vertex id is #1221 and the PR to resolve that issue is #3418 .
Currently JanusGraph supports only `Long` type vertex ids. It would be great to see `ASCII String`, `UUID`, `byte[]` data types to be added as well, so that users have flexibility around what types of ids they chose and how they select them. Some of the users may need it to avoid id block allocation process, some of them may need for data migration purposes, some of them may need it for business use-cases.
As we discussed in #3418 to not have a breaking change for users with `Long` vertex ids (default vertex ids) we introduce an additional byte which determine data type for any key which is different from 8 bytes in size.
Any key which is 8 bytes is considered to be a `Long` data type.
Any other key will have an additional 1 byte at the beginning which tells the next things:
1 bit - reserved to be `1` (needed to determine when the key starts).
2 bits - reserved for latter possible features / extensions.
1 bit - determines if the original key size is less then 8 bits or not.
4 bits - determines data type of the key (potentially we will be able to hold up to 16 data types for vertex id).
Any key with the size equals to 7 bytes is going to be extended to 9 bytes (1 byte is added for data type and 1 dummy byte is added to the end, just because size of 8 bytes is always reserved for `Long` data type). To determine if the 9 bytes key originally was 7 or 8 bytes we use the bit flag explained above (_1 bit - determines if the original key size is less then 8 bits or not._) which is always stored in the first byte of the key.
I believe to cover most of the use-cases the following 4 data types should be enough:
Long - 8 bytes. Only positive numbers. No additional overhead.
#1221 ASCII String - 1 byte per character + 1 byte overhead (to store type). (done in #3418)
UUID - 16 bytes + 1 byte overhead (to store type).
byte[] - 1 byte per byte + 1 byte type overhead + 4 bytes size overhead.
Notice, even so `ASCII String` and `byte[]` could allow using any key size, there are practical limitations on each of the storage / index backends for such keys. Thus the maximum length of the key should be determined by the backends which are used with JanusGraph.
For example, if user uses ElasticSearch as an index backend - vertex ids must be under 512 bytes, do to ElasticSearch limitation of document ids.
Contributor guide
Research direction
Review the related issue #1221 and PR #3418 first, then trace the existing vertex-ID encoding and handling described in this epic. Done means supporting UUID and byte[] vertex IDs while preserving Long compatibility, applying the stated encoding rules and respecting backend key-length limits.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100