JanusGraph / JanusGraph/janusgraph
Support bulk load with native HBase bulk load mechanism
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 5.8k
- Forks
- 1.2k
- Avg merge
- 13h 53m
- Merged PRs (30d)
- 6
Description
Here is the description of the HBase bulk load mechanism:
https://hbase.apache.org/book.html#arch.bulk.load
The idea is to output the data into native HBase data file (hfile) format, and then directly load the data files into HBase instance.
There are a few considerations to convert graph data and bulk load the data.
- Computing framework
1. This most likely will be Spark or MapReduce to generate the HBase data files in parallel. This will be similar to the SparkGraphComputer. Or to a less degree it can be multiple threads to achieve parallelism.
- Janusgraph Ids
1. We will still need a live JanusGraph HBase backend instance. During the process to generate the data files, we will consult the backend to give us the id blocks and ids, the same way we load into the backend directly. But we will write the ids into HBase data files instead.
- Graph schema
1. It is still recommended that the schema is populated into the graph before the bulk load process, and the schema will be consulted during the process to generate the data files.
- Graph indexes
1. We can have two options. The first option is not to generate indexes for the input data during the process, and run batch indexing after the raw data has been bulk loaded into the graph. The second option is to generate the indexes and write to the index backend during the process (which may slow down the process). This is what we can do until we find a bulk load mechanism for the indexes. The composite and vertex-centric indexes do not have this issue since they will be outputted to the hbase data files.
- Source formats
1. Gyro, GraphSON, CSV, GraphML files
2. Hadoop graph input formats, which can be graphs hosted on live instances (e.g JanusGraph CassandraInputFormat, HBaseInputFormat.java). In this case, it will be more like 'Export' and 'Import'.
More ...
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the linked HBase bulk-load mechanism, then read SparkGraphComputer and HBaseInputFormat.java to understand the existing computation and input paths. The scope still needs to define the bulk-load entry point, supported source formats, schema and ID handling, and index behavior before implementation can be considered complete.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, java, spark
- Domain
- backend, data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100