apache / apache/arrow

[Python][Dataset] The first table schema becomes a common schema for the full Dataset

Open
#27,905 6 comments 0 reactions 0 assignees View on GitHub
Component: Documentation Component: Python good-first-issue Status: needs champion Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

The first table schema becomes a common schema for the full Dataset. It could cause problems with sparse data.

Consider example below, when first chunks is full of NA, pyarrow ignores dtypes from pandas for a whole dataset:
```java

# get dataset
!wget https://physionet.org/files/mimiciii-demo/1.4/D_ITEMS.csv

import pandas as pd
import pyarrow.parquet as pq
import pyarrow as pa
import pyarrow.dataset as ds
import shutil
from pathlib import Path

def foo(input_csv='D_ITEMS.csv', output='tmp.parquet', chunksize=1000):
if Path(output).exists():
shutil.rmtree(output) # write dataset
d_items = pd.read_csv(input_csv, index_col='row_id',
usecols=['row_id', 'itemid', 'label', 'dbsource', 'category', 'param_type'],
dtype={'row_id': int, 'itemid': int, 'label': str, 'dbsource': str,
'category': str, 'param_type': str}, chunksize=chunksize) for i, chunk in enumerate(d_items):
table = pa.Table.from_pandas(chunk)
if i == 0:
schema1 = pa.Schema.from_pandas(chunk)
schema2 = table.schema
# print(table.field('param_type'))
pq.write_to_dataset(table, root_path=output)

# read dataset
dataset = ds.dataset(output)

# compare schemas
print('Schemas are equal: ', dataset.schema == schema1 == schema2)
print(dataset.schema.types)
print('Should be string', dataset.schema.field('param_type'))
return dataset
```
```java

dataset = foo()
dataset.to_table()

>>>Schemas are equal: False
[DataType(int64), DataType(string), DataType(string), DataType(null), DataType(null), DataType(int64)]
Should be string pyarrow.Field
---------------------------------------------------------------------------
ArrowTypeError: fields had matching names but differing types. From: category: string To: category: null
```
If you do schemas listing, you'll see that almost all parquet files ignored pandas dtypes:
```java

import os

for i in os.listdir('tmp.parquet/'):
print(ds.dataset(os.path.join('tmp.parquet/', i)).schema.field('param_type'))

>>>pyarrow.Field
pyarrow.Field
pyarrow.Field
pyarrow.Field
pyarrow.Field
pyarrow.Field
pyarrow.Field
pyarrow.Field
pyarrow.Field
pyarrow.Field
pyarrow.Field
pyarrow.Field
pyarrow.Field
```
But if we will get bigger chunk of data, that contains non NA values, everything is OK:
```java

dataset = foo(chunksize=10000)
dataset.to_table()

>>>Schemas are equal: True
[DataType(int64), DataType(string), DataType(string), DataType(string), DataType(string), DataType(int64)]
Should be string pyarrow.Field
pyarrow.Table
itemid: int64
label: string
dbsource: string
category: string
param_type: string
row_id: int64
```
Check NA in data:
```java

pd.read_csv('D_ITEMS.csv', nrows=1000)['param_type'].unique()
>>>array([nan])

pd.read_csv('D_ITEMS.csv', nrows=10000)['param_type'].unique()
>>>array([nan, 'Numeric', 'Text', 'Date time', 'Solution', 'Process',
'Checkbox'], dtype=object)
```
 

 PS: switching issues reporting from github to Jira is outstanding move

 

**Reporter**: [Borys Kabakov](https://issues.apache.org/jira/browse/ARROW-12080)
#### Related issues:
- [The first table schema becomes a common schema for the full Dataset](https://github.com/apache/arrow/issues/27903) (is duplicated by)
- [The first table schema becomes a common schema for the full Dataset](https://github.com/apache/arrow/issues/27904) (is duplicated by)

**Note**: *This issue was originally created as [ARROW-12080](https://issues.apache.org/jira/browse/ARROW-12080). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Reproduce the provided foo() example using pandas, pyarrow.dataset, and pyarrow.parquet.write_to_dataset, comparing sparse and larger chunks. Start at the dataset schema handling shown by ds.dataset(output); done means all chunk schemas preserve the declared string types and dataset.to_table() succeeds without ArrowTypeError.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.