apache / apache/hugegraph-toolchain
[Feature][Umbrella] Support spark for hugegraph-loader module
- Dominant language
- Java
- Stars
- 121
- Forks
- 125
- Avg merge
- 10d 10h
- Merged PRs (30d)
- 3
Description
### Feature Description (功能描述)
Support spark for `hugegraph-loader` module
### Test cmd
The options are divided into two parts, `spark options` and `hugegraph-loader options`. (no sorting required)
```shell
./bin/hugegraph-spark-loader.sh \
--master yarn --deploy-mode client --name spark-test \
--conf spark.eventLog.enabled=false \
--conf spark.executor.extraJavaOptions=-XX:+PrintGCDetails\
-f ./conf/spark.json --username admin --token admin \
-h 127.0.0.1 -p 8093 -g my_graph2
```
### Task list
- [x] #281 @simon824
- [x] #305 @simon824
- [x] #311 @simon824
- [x] #317 @simon824
- [x] https://github.com/apache/incubator-hugegraph-doc/pull/143 @simon824
- [ ] support metrics statistics for spark-loader
- [ ] support create schema for spark-loader
- [ ] date_format option support for spark-loader
- [ ] other options support for spark-loader
- [ ] bugfix
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the bin/hugegraph-spark-loader.sh entry point and conf/spark.json, then run the supplied Spark and hugegraph-loader command. The remaining work covers metrics statistics, schema creation, date_format and other options, plus bug fixes; done means the applicable unchecked tasks are supported and verified with the test command.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, shell, spark
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100