apache / apache/datafusion

DataFrame write api should accept Overwrite option when the file already exist

Aperta
#4,986 2 commenti 1 reazione 1 assegnatario Rivendicata da @Dev79844 Vedi su GitHub
enhancement
Lingua principale
Rust
Stelle
9.3k
Fork
2.4k
Merge medio
3g 11h
PR unite (30g)
360

Descrizione

Problem:
While trying to write a csv (any type) file, I am getting the following exception if the file already exists:
```
Error: Execution("Could not create directory data/processed/diabetes.csv: Os { code: 17, kind: AlreadyExists, message: \"File exists\" }")
```
But other frameworks like Polars, PySpark accept overriding either optional or by default and it also possible to append existing files if selected.

Possible Solution:
I suggest having a `CsvWriteOptions` struct like `CsvReadOptions` struct which must be passed during writing the file. For example `df.write_csv(path: "my/precious/path", CsvWriteOptions::default()).await?` This family of write options can also have all the configuration possibilities for json, parquet, csv, table etc. For example, If I want to write the csv into multiple file or one single file.

Also not sure why by default csv file is written into multiple partition `part01.csv ... part08.csv` by default, I should let user write into one csv file incase of smaller dataset.

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.