apache / apache/hugegraph-toolchain

[Feature][Umbrella] Support spark for hugegraph-loader module

Open
#283 3 comments 3 reactions 0 assignees View on GitHub
feature
Dominant language
Java
Stars
121
Forks
125
Avg merge
10d 10h
Merged PRs (30d)
3

Description

### Feature Description (功能描述)

Support spark for `hugegraph-loader` module

### Test cmd
The options are divided into two parts, `spark options` and `hugegraph-loader options`. (no sorting required)
```shell
./bin/hugegraph-spark-loader.sh \
--master yarn --deploy-mode client --name spark-test \
--conf spark.eventLog.enabled=false \
--conf spark.executor.extraJavaOptions=-XX:+PrintGCDetails\
-f ./conf/spark.json --username admin --token admin \
-h 127.0.0.1 -p 8093 -g my_graph2
```

### Task list
- [x] #281 @simon824
- [x] #305 @simon824
- [x] #311 @simon824
- [x] #317 @simon824
- [x] https://github.com/apache/incubator-hugegraph-doc/pull/143 @simon824
- [ ] support metrics statistics for spark-loader
- [ ] support create schema for spark-loader
- [ ] date_format option support for spark-loader
- [ ] other options support for spark-loader
- [ ] bugfix

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the bin/hugegraph-spark-loader.sh entry point and conf/spark.json, then run the supplied Spark and hugegraph-loader command. The remaining work covers metrics statistics, schema creation, date_format and other options, plus bug fixes; done means the applicable unchecked tasks are supported and verified with the test command.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, shell, spark
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.