[Python][Dataset] The first table schema becomes a common schema for the full Dataset
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
The first table schema becomes a common schema for the full Dataset. It could cause problems with sparse data.
Consider example below, when first chunks is full of NA, pyarrow ignores dtypes from pandas for a whole dataset:
```java
# get dataset
!wget https://physionet.org/files/mimiciii-demo/1.4/D_ITEMS.csv
import pandas as pd
import pyarrow.parquet as pq
import pyarrow as pa
import pyarrow.dataset as ds
import shutil
from pathlib import Path
def foo(input_csv='D_ITEMS.csv', output='tmp.parquet', chunksize=1000):
if Path(output).exists():
shutil.rmtree(output) # write dataset
d_items = pd.read_csv(input_csv, index_col='row_id',
usecols=['row_id', 'itemid', 'label', 'dbsource', 'category', 'param_type'],
dtype={'row_id': int, 'itemid': int, 'label': str, 'dbsource': str,
'category': str, 'param_type': str}, chunksize=chunksize) for i, chunk in enumerate(d_items):
table = pa.Table.from_pandas(chunk)
if i == 0:
schema1 = pa.Schema.from_pandas(chunk)
schema2 = table.schema
# print(table.field('param_type'))
pq.write_to_dataset(table, root_path=output)
# read dataset
dataset = ds.dataset(output)
# compare schemas
print('Schemas are equal: ', dataset.schema == schema1 == schema2)
print(dataset.schema.types)
print('Should be string', dataset.schema.field('param_type'))
return dataset
```
```java
dataset = foo()
dataset.to_table()
>>>Schemas are equal: False
[DataType(int64), DataType(string), DataType(string), DataType(null), DataType(null), DataType(int64)]
Should be string pyarrow.Field
---------------------------------------------------------------------------
ArrowTypeError: fields had matching names but differing types. From: category: string To: category: null
```
If you do schemas listing, you'll see that almost all parquet files ignored pandas dtypes:
```java
import os
for i in os.listdir('tmp.parquet/'):
print(ds.dataset(os.path.join('tmp.parquet/', i)).schema.field('param_type'))
>>>pyarrow.Field
pyarrow.Field
pyarrow.Field
pyarrow.Field
pyarrow.Field
pyarrow.Field
pyarrow.Field
pyarrow.Field
pyarrow.Field
pyarrow.Field
pyarrow.Field
pyarrow.Field
pyarrow.Field
```
But if we will get bigger chunk of data, that contains non NA values, everything is OK:
```java
dataset = foo(chunksize=10000)
dataset.to_table()
>>>Schemas are equal: True
[DataType(int64), DataType(string), DataType(string), DataType(string), DataType(string), DataType(int64)]
Should be string pyarrow.Field
pyarrow.Table
itemid: int64
label: string
dbsource: string
category: string
param_type: string
row_id: int64
```
Check NA in data:
```java
pd.read_csv('D_ITEMS.csv', nrows=1000)['param_type'].unique()
>>>array([nan])
pd.read_csv('D_ITEMS.csv', nrows=10000)['param_type'].unique()
>>>array([nan, 'Numeric', 'Text', 'Date time', 'Solution', 'Process',
'Checkbox'], dtype=object)
```
PS: switching issues reporting from github to Jira is outstanding move
**Reporter**: [Borys Kabakov](https://issues.apache.org/jira/browse/ARROW-12080)
#### Related issues:
- [The first table schema becomes a common schema for the full Dataset](https://github.com/apache/arrow/issues/27903) (is duplicated by)
- [The first table schema becomes a common schema for the full Dataset](https://github.com/apache/arrow/issues/27904) (is duplicated by)
**Note**: *This issue was originally created as [ARROW-12080](https://issues.apache.org/jira/browse/ARROW-12080). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Reproduce the provided foo() example using pandas, pyarrow.dataset, and pyarrow.parquet.write_to_dataset, comparing sparse and larger chunks. Start at the dataset schema handling shown by ds.dataset(output); done means all chunk schemas preserve the declared string types and dataset.to_table() succeeds without ArrowTypeError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100