JanusGraph / JanusGraph/janusgraph

Support bulk load with native HBase bulk load mechanism

Open
#885 1 comment 10 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

kind/enhancement storage/hbase
Dominant language
Java
Stars
5.8k
Forks
1.2k
Avg merge
13h 53m
Merged PRs (30d)
6

Description

Here is the description of the HBase bulk load mechanism:
https://hbase.apache.org/book.html#arch.bulk.load

The idea is to output the data into native HBase data file (hfile) format, and then directly load the data files into HBase instance.

There are a few considerations to convert graph data and bulk load the data.
- Computing framework
1. This most likely will be Spark or MapReduce to generate the HBase data files in parallel. This will be similar to the SparkGraphComputer. Or to a less degree it can be multiple threads to achieve parallelism.

- Janusgraph Ids
1. We will still need a live JanusGraph HBase backend instance. During the process to generate the data files, we will consult the backend to give us the id blocks and ids, the same way we load into the backend directly. But we will write the ids into HBase data files instead.

- Graph schema
1. It is still recommended that the schema is populated into the graph before the bulk load process, and the schema will be consulted during the process to generate the data files.

- Graph indexes
1. We can have two options. The first option is not to generate indexes for the input data during the process, and run batch indexing after the raw data has been bulk loaded into the graph. The second option is to generate the indexes and write to the index backend during the process (which may slow down the process). This is what we can do until we find a bulk load mechanism for the indexes. The composite and vertex-centric indexes do not have this issue since they will be outputted to the hbase data files.

- Source formats
1. Gyro, GraphSON, CSV, GraphML files
2. Hadoop graph input formats, which can be graphs hosted on live instances (e.g JanusGraph CassandraInputFormat, HBaseInputFormat.java). In this case, it will be more like 'Export' and 'Import'.

More ...

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the linked HBase bulk-load mechanism, then read SparkGraphComputer and HBaseInputFormat.java to understand the existing computation and input paths. The scope still needs to define the bulk-load entry point, supported source formats, schema and ID handling, and index behavior before implementation can be considered complete.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, java, spark
Domain
backend, data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.