apache / apache/datafusion-comet
Comet writer doesn't respect `OVERWRITE` mode
Open
area:writer
bug
priority:medium
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### Describe the bug
Comet fails if the folder exists and cannot overwrite it
### Steps to reproduce
Folder `output2` exists
```
spark.read.parquet("/tmp/writertest/input").repartition(1).write.mode("overwrite").parquet("/tmp/writertest/output2")
25/12/23 08:31:40 ERROR CometExecIterator: Native execution for task 17 failed
org.apache.comet.CometNativeException: Failed to create output file '/tmp/writertest/output2/part-00000-00017.parquet': Is a directory (os error 21)
```
### Expected behavior
Should overwrite the folder with new content
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.