apache / apache/arrow

[Python] Mismatching schema when writing dataset

Open
#14,763 2 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: usage
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the usage question you have. Please include as many useful details as possible.

write_to_dataset the dataframe and report the following error:
```
File "pyarrow\_dataset.pyx", line 476, in pyarrow._dataset.InMemoryDataset.__init__
pyarrow.lib.ArrowTypeError: Item has schema
issuer_code: string
industry_code: string
sector_code: string
sector_name: string
industry_name: string
trade_date: string
-- schema metadata --
pandas: '{"index_columns": ["trade_date"], "column_indexes": [{"name": nu' + 944
which does not match expected schema
issuer_code: string
-- field metadata --
table_field: 'PARTY_CODE'
trade_date: string
-- field metadata --
table_field: 'START_DATE'
industry_code: string
-- field metadata --
table_field: 'INDU_SYS_CODE'
sector_code: string
-- field metadata --
table_field: 'INDU_CODE_1ST'
sector_name: string
-- field metadata --
table_field: 'INDU_NAME_1ST'
industry_name: string
-- field metadata --
table_field: 'PARA_NAME'
-- schema metadata --
issuer_code: '主体编码'
trade_date: '变动起始日'
industry_code: '行业分类体系内部编码'
sector_code: '一级行业分类代码'
sector_name: '一级行业分类名称'
industry_name: '常量名称'

Process finished with exit code 1

```

print(df.index)
```
Index(['20160817', '20100318', '20110825', '20120313', '20120612', '20120720',
'20120823', '20121017', '20140417', '20140425',
...
'20221111', '20221111', '20221111', '20221111', '20221111', '20221111',
'20221111', '20221111', '20221111', '20221111'],
dtype='object', name='trade_date', length=439691)
```

I found it because I use a column in the dataframe as an index, and if I want to keep that index, how to define pa.schema in this case?

The currently defined schema is as follows:
```
schema = pa.schema([
pa.field(Dimension.ISSUER_CODE, pa.string(), metadata={b'table_field': b'PARTY_CODE'}),
pa.field(Dimension.TRADE_DATE, pa.string(), metadata={b'table_field': b'START_DATE'}),
pa.field(Dimension.INDUSTRY_CODE, pa.string(), metadata={b'table_field': b'INDU_SYS_CODE'}),
pa.field(Dimension.SECTOR_CODE, pa.string(), metadata={b'table_field': b'INDU_CODE_1ST'}),
pa.field(Dimension.SECTOR_NAME, pa.string(), metadata={b'table_field': b'INDU_NAME_1ST'}),
pa.field(Dimension.INDUSTRY_NAME, pa.string(), metadata={b'table_field': b'PARA_NAME'}),
],
metadata={
Dimension.ISSUER_CODE: '主体编码',
Dimension.TRADE_DATE: '变动起始日',
Dimension.INDUSTRY_CODE: '行业分类体系内部编码',
Dimension.SECTOR_CODE: '一级行业分类代码',
Dimension.SECTOR_NAME: '一级行业分类名称',
Dimension.INDUSTRY_NAME: '常量名称',
}
)
```

### Component(s)

Parquet, Python

Contributor guide

Open the contributing guide

Research direction

Start at the write_to_dataset entry point and pyarrow._dataset.pyx around InMemoryDataset.__init__, using the reported dataframe index and pa.schema definition as the reproduction. Determine how the index and field metadata are compared, then document or test the schema needed for writing the indexed dataframe successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.