dask / dask/dask

Output files with .csv file extension instead of .part when to_csv is used

Open
#9,044 9 comments 2 reactions 0 assignees View on GitHub
dataframe enhancement good second issue
Dominant language
Python
Stars
13.9k
Forks
2k
PR merge metrics
No merged PRs in 30d

Description

The `to_csv` method outputs filenames with a `.part` extension by default. This post argues that `to_csv` should output CSV files with a `.csv` extension by default.

Let's create a DataFrame and write it out to a directory and see the default behavior.

```python
import pandas as pd
import dask.dataframe as dd

pdf = pd.DataFrame(
{"num1": [1, 2, 3, 4], "num2": [7, 8, 9, 10]},
)

ddf = dd.from_pandas(pdf, npartitions=2)

ddf.to_csv("./data/csv_simple")
```

Here are the files that are output:

```
csv_simple/
0.part
1.part
```

Downstream readers need to do something like `dd.read_csv("./data/csv_simple/*.part")` to read these CSV files into a DataFrame.

Here's the work-around to output the files with the `.csv` file extension:

```python
ddf.to_csv("./data/csv_simple2/whatever-*-hi.csv")
```

Here are the files that are output:

```
csv_simple2/
whatever-0-hi.csv
whatever-1-hi.csv
```

I don't believe there is a work around to customize the file extension and avoid writing `.part` when you use the `name_function` argument. Take a look at this example:

```python
ddf.to_csv("./data/csv_simple3", name_function = lambda x: f"i-like-{x}.csv")
```

Here's what's output:

```
csv_simple3/
i-like-0.csv.part
i-like-1.csv.part
```

I'd personally prefer for files to be written with a `.csv` extension by default. That'd be more intuitive for me.

Think it's also more consistent with the Parquet writers. For example `ddf.to_parquet("./data/parquet_simple")` outputs files like `part.0.parquet` and `part.1.parquet`. Let me know what you think!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.