JanusGraph / JanusGraph/janusgraph

Support ASCII String, UUID, byte[] custom vertex ids

Open
#3,732 1 comment 1 reaction 0 assignees View on GitHub
area/core kind/enhancement
Dominant language
Java
Stars
5.8k
Forks
1.2k
Avg merge
13h 53m
Merged PRs (30d)
6

Description

This is an Epic issue to add multiple different possible data types as vertex ids into JanusGraph.

The issue regarding adding custom String vertex id is #1221 and the PR to resolve that issue is #3418 .

Currently JanusGraph supports only `Long` type vertex ids. It would be great to see `ASCII String`, `UUID`, `byte[]` data types to be added as well, so that users have flexibility around what types of ids they chose and how they select them. Some of the users may need it to avoid id block allocation process, some of them may need for data migration purposes, some of them may need it for business use-cases.

As we discussed in #3418 to not have a breaking change for users with `Long` vertex ids (default vertex ids) we introduce an additional byte which determine data type for any key which is different from 8 bytes in size.
Any key which is 8 bytes is considered to be a `Long` data type.
Any other key will have an additional 1 byte at the beginning which tells the next things:
1 bit - reserved to be `1` (needed to determine when the key starts).
2 bits - reserved for latter possible features / extensions.
1 bit - determines if the original key size is less then 8 bits or not.
4 bits - determines data type of the key (potentially we will be able to hold up to 16 data types for vertex id).

Any key with the size equals to 7 bytes is going to be extended to 9 bytes (1 byte is added for data type and 1 dummy byte is added to the end, just because size of 8 bytes is always reserved for `Long` data type). To determine if the 9 bytes key originally was 7 or 8 bytes we use the bit flag explained above (_1 bit - determines if the original key size is less then 8 bits or not._) which is always stored in the first byte of the key.

I believe to cover most of the use-cases the following 4 data types should be enough:
Long - 8 bytes. Only positive numbers. No additional overhead.
#1221 ASCII String - 1 byte per character + 1 byte overhead (to store type). (done in #3418)
UUID - 16 bytes + 1 byte overhead (to store type).
byte[] - 1 byte per byte + 1 byte type overhead + 4 bytes size overhead.

Notice, even so `ASCII String` and `byte[]` could allow using any key size, there are practical limitations on each of the storage / index backends for such keys. Thus the maximum length of the key should be determined by the backends which are used with JanusGraph.
For example, if user uses ElasticSearch as an index backend - vertex ids must be under 512 bytes, do to ElasticSearch limitation of document ids.

Contributor guide

Open the contributing guide

Research direction

Review the related issue #1221 and PR #3418 first, then trace the existing vertex-ID encoding and handling described in this epic. Done means supporting UUID and byte[] vertex IDs while preserving Long compatibility, applying the stated encoding rules and respecting backend key-length limits.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.