[Python] Mismatching schema when writing dataset
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the usage question you have. Please include as many useful details as possible.
write_to_dataset the dataframe and report the following error:
```
File "pyarrow\_dataset.pyx", line 476, in pyarrow._dataset.InMemoryDataset.__init__
pyarrow.lib.ArrowTypeError: Item has schema
issuer_code: string
industry_code: string
sector_code: string
sector_name: string
industry_name: string
trade_date: string
-- schema metadata --
pandas: '{"index_columns": ["trade_date"], "column_indexes": [{"name": nu' + 944
which does not match expected schema
issuer_code: string
-- field metadata --
table_field: 'PARTY_CODE'
trade_date: string
-- field metadata --
table_field: 'START_DATE'
industry_code: string
-- field metadata --
table_field: 'INDU_SYS_CODE'
sector_code: string
-- field metadata --
table_field: 'INDU_CODE_1ST'
sector_name: string
-- field metadata --
table_field: 'INDU_NAME_1ST'
industry_name: string
-- field metadata --
table_field: 'PARA_NAME'
-- schema metadata --
issuer_code: '主体编码'
trade_date: '变动起始日'
industry_code: '行业分类体系内部编码'
sector_code: '一级行业分类代码'
sector_name: '一级行业分类名称'
industry_name: '常量名称'
Process finished with exit code 1
```
print(df.index)
```
Index(['20160817', '20100318', '20110825', '20120313', '20120612', '20120720',
'20120823', '20121017', '20140417', '20140425',
...
'20221111', '20221111', '20221111', '20221111', '20221111', '20221111',
'20221111', '20221111', '20221111', '20221111'],
dtype='object', name='trade_date', length=439691)
```
I found it because I use a column in the dataframe as an index, and if I want to keep that index, how to define pa.schema in this case?
The currently defined schema is as follows:
```
schema = pa.schema([
pa.field(Dimension.ISSUER_CODE, pa.string(), metadata={b'table_field': b'PARTY_CODE'}),
pa.field(Dimension.TRADE_DATE, pa.string(), metadata={b'table_field': b'START_DATE'}),
pa.field(Dimension.INDUSTRY_CODE, pa.string(), metadata={b'table_field': b'INDU_SYS_CODE'}),
pa.field(Dimension.SECTOR_CODE, pa.string(), metadata={b'table_field': b'INDU_CODE_1ST'}),
pa.field(Dimension.SECTOR_NAME, pa.string(), metadata={b'table_field': b'INDU_NAME_1ST'}),
pa.field(Dimension.INDUSTRY_NAME, pa.string(), metadata={b'table_field': b'PARA_NAME'}),
],
metadata={
Dimension.ISSUER_CODE: '主体编码',
Dimension.TRADE_DATE: '变动起始日',
Dimension.INDUSTRY_CODE: '行业分类体系内部编码',
Dimension.SECTOR_CODE: '一级行业分类代码',
Dimension.SECTOR_NAME: '一级行业分类名称',
Dimension.INDUSTRY_NAME: '常量名称',
}
)
```
### Component(s)
Parquet, Python
Contributor guide
Research direction
Start at the write_to_dataset entry point and pyarrow._dataset.pyx around InMemoryDataset.__init__, using the reported dataframe index and pa.schema definition as the reproduction. Determine how the index and field metadata are compared, then document or test the schema needed for writing the indexed dataframe successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100