apache / apache/incubator-graphar

[Bug][Spark]: `overwrite` mode not work with GarDataSource

Open
#333 0 comments 0 reactions 1 assignee Claimed by @acezen View on GitHub
bug
Dominant language
C++
Stars
373
Forks
93
Avg merge
7d 21h
Merged PRs (30d)
7

Description

### Is there an existing issue for this?

- [X] I have searched the existing issues

### Current Behavior

If GraphAr try to write dataframe with `overwrite` mode and `GarDataSource` like

```java
dataFrame.write
.mode("overwrite")
.option("header", "true")
.option("fileFormat", fileType)
.option(
GeneralParams.offsetStartChunkIndexKey,
offsetStartChunkIndex.get
)
.format("com.alibaba.graphar.datasources.GarDataSource")
.save(outputPrefix)
```
it would raise error:
```bash
org.apache.spark.sql.AnalysisException: Table gar file:/tmp/test1/edge/person_knows_person/ordered_by_source/offset does not support truncate in batch mode.;
OverwriteByExpression RelationV2[_graphArOffset#339] gar file:/tmp/test1/edge/person_knows_person/ordered_by_source/offset, true, [header=true, fileFormat=csv, _graphar_offset_start_chunk_index=0, path=/tmp/test1/edge/person_knows_person/ordered_by_source/offset/], true
+- LogicalRDD [_graphArOffset#216], false
at org.apache.spark.sql.errors.QueryCompilationErrors$.unsupportedTableOperationError(QueryCompilationErrors.scala:801)
at org.apache.spark.sql.errors.QueryCompilationErrors$.unsupportedTruncateInBatchModeError(QueryCompilationErrors.scala:821)
at org.apache.spark.sql.execution.datasources.v2.TableCapabilityCheck$.$anonfun$apply$1(TableCapabilityCheck.scala:61)
at org.apache.spark.sql.execution.datasources.v2.TableCapabilityCheck$.$anonfun$apply$1$adapted(TableCapabilityCheck.scala:40)
at org.apache.spark.sql.catalyst.trees.TreeNode.foreach(TreeNode.scala:253)
at org.apache.spark.sql.execution.datasources.v2.TableCapabilityCheck$.apply(TableCapabilityCheck.scala:40)
at org.apache.spark.sql.execution.datasources.v2.TableCapabilityCheck$.apply(TableCapabilityCheck.scala:32)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis$39(CheckAnalysis.scala:560)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis$39$adapted(CheckAnalysis.scala:560)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
```

### Expected Behavior

write with `overwrite` as expect.

### Minimal Reproducible Example

NA

### Environment

- Operating system: macOS M2
- GraphAr version: 0.11.0

### Link to GraphAr Logs

_No response_

### Further Information

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.