apache / apache/incubator-graphar
[Feat][Spark] Improve the compatibility of the GraphAr Spark library
- Dominant language
- C++
- Stars
- 373
- Forks
- 93
- Avg merge
- 7d 21h
- Merged PRs (30d)
- 7
Description
**Is your feature request related to a problem? Please describe.**
The APIs of Spark (especially DataFrame reader and writer) is different between 2.x and 3.x. Currently, the GraphAr spark library is developed based on spark 3.x, and we should make it compatible with spark 2.x.
**Describe the solution you'd like**
The spark library code directories could be:
- `common`: the common code that could be run in 2.x and 3.x both;
- `spark-2.x`: the code for spark 2.x;
- `spark-3.x`: the code for spark 3.x.
The users can build packages for spark 2.x and 3.x separately.
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
**Additional context**
Add any other context or screenshots about the feature request here.
Contributor guide
Research direction
Start by locating the current GraphAr Spark library code and the DataFrame reader and writer API usage. Compare the compatibility requirements for Spark 2.x and 3.x, then organize common, spark-2.x, and spark-3.x code so users can build separate packages for each version.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- spark
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100