apache / apache/arrow

[Python][Parquet] Cannot read parquet with duplicate column names

Open
#37,428 6 comments 0 reactions 0 assignees View on GitHub
Component: Parquet Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

If `use_legacy_dataset=False` in `use_legacy_dataset` then Exception is raised.
If we specify the `schema`, the read succeeds, but the values of the duplicate columns are `null`.

Using `use_legacy_dataset=True` there are no issues reading the single file (but we can only `rename_columns` after reading the whole file).

Alternatively the `columns` param of `read_table` should allow reading using index, not just column names.

### Component(s)

Parquet, Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.