apache / apache/incubator-graphar

Scalable conversion to GraphAr format

Open
#917 1 comment 2 reactions 0 assignees View on GitHub
enhancement
Dominant language
C++
Stars
373
Forks
93
Avg merge
7d 21h
Merged PRs (30d)
7

Description

### Describe the enhancement requested

Today the conversion/import path does not seem to scale well for larger datasets.

From reading the current code and trying it in practice, I see that:
- the C++ high-level builders are convenience APIs and keep data in memory until `Dump()`
- the Spark writer scales better in principle, but still does heavy batch work such as index generation, joins, sorting, repartitioning, and offset construction

Because GraphAr is positioned for use with "large-scale graph data", it would be useful for a community to have a clearer path for scalable conversion.

Assuming I'm not missing something, my suggestion is:
- keep the C++ high-level writer/builder path simple/reference-oriented and convenient for small/medium imports
- optimize the Spark API/writer as the primary path for large-scale conversion

This way we treat Spark as the practical scalable backend for data lake, object stores, HDFS, and distributed preprocessing.

Why Spark seems like the better place to optimize first:
- Spark is considered a data lake first-class citizen, used by many orgs in production and thus in practice is more accessible for end-users (compared to a dedicated VM only for C++ import)
- storage backends such as S3/HDFS are abstracted through Spark/Hadoop
- large joins / remapping / repartitioning are natural Spark workloads
- avoiding two separate “fully optimized” implementations (Spark and Cpp) may be easier to maintain long-term

To sum up, would the project agree with this direction?

If it sounds reasonable, I’d be happy to help investigate and propose concrete improvements in the Spark conversion path.

### Component(s)

C++, Spark

Contributor guide

Open the contributing guide

Research direction

Start by tracing the C++ high-level builders through Dump() and reviewing the Spark writer's index generation, joins, sorting, repartitioning, and offset construction. Determine which Spark conversion bottlenecks are measurable and define a concrete optimization scope. Done should include an agreed scalable conversion direction and corresponding evidence of improvement.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, spark
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.