to_csv save mode default & options
- Dominant language
- Python
- Stars
- 13.9k
- Forks
- 2k
- PR merge metrics
- No merged PRs in 30d
Description
This issue summarizes the current behavior of the `mode` flag in `to_csv`, when it behaves unexpectedly, and what options users are likely to want.
The default `to_csv` save mode is "wt", but the docs say "w", so that's a little typo to fix.
pandas has different `to_csv` write modes like w+, w, and a.
Dask `to_csv` uses fsspec `open_files` under the hood, which has write modes like ‘rb’, ‘wt’, etc.
It's hard to decipher the exhaustive list of write modes in the [pandas docs](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_csv.html), [fsspec docs](https://filesystem-spec.readthedocs.io/en/latest/api.html), and [Dask docs](https://docs.dask.org/en/stable/generated/dask.dataframe.to_csv.html). It doesn't seem like any of the docs are providing complete lists.
My first goal is to create the list of all the write modes available to Dask users and get them in the docs. Dask users can presumably use the "rb" write mode, but I'm not actually sure what that stands for.
I did some experimentation and here's the behavior of the save modes I'm observing. Let's start by writing some data.
```python
import dask.dataframe as dd
import pandas as pd
df = pd.DataFrame(
{"num1": [1, 2, 3, 4], "num2": [7, 8, 9, 10]},
)
df = dd.from_pandas(df, npartitions=2)
df.to_csv("csv_danger")
```
This will output the following files:
```
csv_danger/
0.part
1.part
```
We cannot append to a directory so the following code will correctly error out with `IsADirectoryError`:
```python
df = pd.DataFrame(
{"firstname": ["cat"], "lastname": ["dog"]},
)
ddf = dd.from_pandas(df, npartitions=1)
ddf.to_csv("csv_danger", mode="a")
```
We can append to a file:
```python
df = pd.DataFrame(
{"num1": [10, 20], "num2": [70, 80]},
)
df = dd.from_pandas(df, npartitions=1)
df.to_csv("csv_danger/0.part", mode="a")
```
Appending to a file probably only works on localhost and not in a cloud storage system. I suppose that's alright.
Now let's try `w+` save mode:
```python
df = pd.DataFrame(
{"num1": [10, 20], "num2": [70, 80]},
)
df = dd.from_pandas(df, npartitions=1)
df.to_csv("csv_danger", mode="w+")
```
`w+` save mode will overwrite `0.part` and leave `1.part` unchanged. I would have expected for all the files in the `csv_danger` to get deleted.
Let's try `w` save mode.
```python
df = pd.DataFrame(
{"num1": [88, 99], "num2": [2, 3]},
)
df = dd.from_pandas(df, npartitions=1)
df.to_csv("csv_danger", mode="w")
```
This also overwrites `0.part` and leaves `1.part` unchanged.
Let's try `wt` save mode, which is the default:
```python
df = pd.DataFrame(
{"num1": [66, 77], "num2": [5, 6]},
)
df = dd.from_pandas(df, npartitions=1)
df.to_csv("csv_danger", mode="wt")
```
This also only overwrites `0.part` and doesn't touch `1.part`.
As a user, here are the save modes I would like:
* overwrite: delete all the existing files and perform a write with new data
* append: append files to the existing directory (without touching the existing files). Append is a bad word here because Dask uses append to indicate appending data to an existing file. Not sure what to call this. I don't want filenames to conflict when adding additional files to a folder.
* errorifexists: if the directory already contains data, then error out and don't perform a write
I don't personally ever want the current Dask default behavior which is overwriting some files and leaving the others untouched. The "partial clobber" write mode is something I'd like to avoid.
Next steps:
* Let's figure out all the current write mode options & document their behavior
* Let's decide on if other write modes should be exposed to users.
* Let's think about how we can be compatible with other Dask writer APIs. Users can run `ddf.to_parquet("some/dir", overwrite=True)` for example.
My preference will probably be to add a separate option to all writers with consistent options. `ddf.to_csv("something", save_mode="overwrite")` and `ddf.to_parquet("something", save_mode="overwrite")` would be nice. The pandas writer APIs aren't consistent so they'll be of limited use here (`pandas.DataFrame.to_parquet`, `pandas.DataFrame.to_json`, and `pandas.DataFrame.to_csv` don't have a unified `save_mode` with options that are consistent throughout).
Contributor guide
Assessment
This issue has not been assessed yet.