apache / apache/incubator-graphar

[Feat][Spark] Improve the compatibility of the GraphAr Spark library

Open
#99 0 comments 0 reactions 0 assignees View on GitHub
enhancement priority:low
Dominant language
C++
Stars
373
Forks
93
Avg merge
7d 21h
Merged PRs (30d)
7

Description

**Is your feature request related to a problem? Please describe.**
The APIs of Spark (especially DataFrame reader and writer) is different between 2.x and 3.x. Currently, the GraphAr spark library is developed based on spark 3.x, and we should make it compatible with spark 2.x.

**Describe the solution you'd like**
The spark library code directories could be:
- `common`: the common code that could be run in 2.x and 3.x both;
- `spark-2.x`: the code for spark 2.x;
- `spark-3.x`: the code for spark 3.x.

The users can build packages for spark 2.x and 3.x separately.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.

Contributor guide

Open the contributing guide

Research direction

Start by locating the current GraphAr Spark library code and the DataFrame reader and writer API usage. Compare the compatibility requirements for Spark 2.x and 3.x, then organize common, spark-2.x, and spark-3.x code so users can build separate packages for each version.

Written by the indexing model from the issue text.

Assessment

Tech stack
spark
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.