apache / apache/arrow

[Python] Parquet write_to_dataset leads to partial write when unsupported datatype is passed in table

Open
#30,859 3 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

Trying to save unsupported datatype in parquet using pyarrow.write_to_dataset results in a partial folder and file write to disk.

 
```java

import pandas as pd
import numpy as np
import pyarrow as pa
import pyarrow.parquet as pq

data = np.arange(2, 10, dtype=np.float16)
df = pd.DataFrame(data=data, columns=['fp16'])
table=pa.Table.from_pandas(df)

pq.write_to_dataset(table=table, root_path='./fp16_fail_dataset')
```
 

**Environment**: Linux (Ubuntu 20.04)
**Reporter**: [Chandrasekaran Anirudh Bhardwaj](https://issues.apache.org/jira/browse/ARROW-15375)
#### Externally tracked issue: [https://github.com/pandas-dev/pandas/pull/44847](https://github.com/pandas-dev/pandas/pull/44847)

**Note**: *This issue was originally created as [ARROW-15375](https://issues.apache.org/jira/browse/ARROW-15375). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Run the provided Python reproduction, starting at pyarrow.parquet.write_to_dataset with the float16 pandas table. Confirm the partial folder and file writes, then review the externally tracked pandas PR for the intended handling; done means an unsupported datatype does not leave partial output on disk.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, pandas, python
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.