apache / apache/incubator-graphar
[Bug][Spark]: `GarDataSource` format does't support parallel write multiple dataframe
- Dominant language
- C++
- Stars
- 373
- Forks
- 93
- Avg merge
- 7d 21h
- Merged PRs (30d)
- 7
Description
### Is there an existing issue for this?
- [X] I have searched the existing issues
### Current Behavior
When try to write multiple data frame like:
```java
val offsetChunks = edgeDfAndOffsetDf._2.par
offsetChunks.foreach { case (i, offsetChunk) =>
FileSystem.writeDataFrame(
offsetChunk,
FileType.FileTypeToString(fileType),
outputPrefix,
Some(i),
None
)
offsetChunk.unpersist()
}
```
got error
### Expected Behavior
support parallel write.
### Minimal Reproducible Example
NA
### Environment
- Operating system:macOS M2
- GraphAr version: v0.11.0
### Link to GraphAr Logs
_No response_
### Further Information
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.