Output files with .csv file extension instead of .part when to_csv is used
- Dominant language
- Python
- Stars
- 13.9k
- Forks
- 2k
- PR merge metrics
- No merged PRs in 30d
Description
The `to_csv` method outputs filenames with a `.part` extension by default. This post argues that `to_csv` should output CSV files with a `.csv` extension by default.
Let's create a DataFrame and write it out to a directory and see the default behavior.
```python
import pandas as pd
import dask.dataframe as dd
pdf = pd.DataFrame(
{"num1": [1, 2, 3, 4], "num2": [7, 8, 9, 10]},
)
ddf = dd.from_pandas(pdf, npartitions=2)
ddf.to_csv("./data/csv_simple")
```
Here are the files that are output:
```
csv_simple/
0.part
1.part
```
Downstream readers need to do something like `dd.read_csv("./data/csv_simple/*.part")` to read these CSV files into a DataFrame.
Here's the work-around to output the files with the `.csv` file extension:
```python
ddf.to_csv("./data/csv_simple2/whatever-*-hi.csv")
```
Here are the files that are output:
```
csv_simple2/
whatever-0-hi.csv
whatever-1-hi.csv
```
I don't believe there is a work around to customize the file extension and avoid writing `.part` when you use the `name_function` argument. Take a look at this example:
```python
ddf.to_csv("./data/csv_simple3", name_function = lambda x: f"i-like-{x}.csv")
```
Here's what's output:
```
csv_simple3/
i-like-0.csv.part
i-like-1.csv.part
```
I'd personally prefer for files to be written with a `.csv` extension by default. That'd be more intuitive for me.
Think it's also more consistent with the Parquet writers. For example `ddf.to_parquet("./data/parquet_simple")` outputs files like `part.0.parquet` and `part.1.parquet`. Let me know what you think!
Contributor guide
Assessment
This issue has not been assessed yet.