apache / apache/incubator-graphar

[Bug][Spark]: `GarDataSource` format does't support parallel write multiple dataframe

Open
#334 0 comments 0 reactions 1 assignee Claimed by @acezen View on GitHub
bug
Dominant language
C++
Stars
373
Forks
93
Avg merge
7d 21h
Merged PRs (30d)
7

Description

### Is there an existing issue for this?

- [X] I have searched the existing issues

### Current Behavior

When try to write multiple data frame like:
```java
val offsetChunks = edgeDfAndOffsetDf._2.par
offsetChunks.foreach { case (i, offsetChunk) =>
FileSystem.writeDataFrame(
offsetChunk,
FileType.FileTypeToString(fileType),
outputPrefix,
Some(i),
None
)
offsetChunk.unpersist()
}
```
got error
截屏2024-01-19 15 24 29

### Expected Behavior

support parallel write.

### Minimal Reproducible Example

NA

### Environment

- Operating system:macOS M2
- GraphAr version: v0.11.0

### Link to GraphAr Logs

_No response_

### Further Information

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.