apache / apache/arrow

[C++][Dataset] Handling of duplicate columns in Dataset factory and scanning

Open
#24,407 4 comments 0 reactions 0 assignees View on GitHub
Component: C++ Priority: Major Status: needs champion Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

While testing duplicate column names, I ran into multiple issues:

- Factory fails if there are duplicate columns, even for a single file
- In addition, we should also fix and/or test that factory works for duplicate columns if the schema's are equal
- Once a Dataset with duplicated columns is created, scanning without any column projection fails

My python reproducer:

```python

import pyarrow as pa
import pyarrow.parquet as pq
import pyarrow.dataset as ds
import pyarrow.fs

# create single parquet file with duplicated column names
table = pa.table([pa.array([1, 2, 3]), pa.array([4, 5, 6]), pa.array([7, 8, 9])], names=['a', 'b', 'a'])
pq.write_table(table, "data_duplicate_columns.parquet")
```

Factory fails:

```Java

dataset = ds.dataset("data_duplicate_columns.parquet", format="parquet")
...
~/scipy/repos/arrow/python/pyarrow/dataset.py in dataset(paths_or_factories, filesystem, partitioning, format)
346
347 factories = [_ensure_factory(f, **kwargs) for f in paths_or_factories]
--> 348 return UnionDatasetFactory(factories).finish()
349
350

ArrowInvalid: Can't unify schema with duplicate field names.
```

And when creating a Dataset manually:

```python

schema = pa.schema([('a', 'int64'), ('b', 'int64'), ('a', 'int64')])
dataset = ds.FileSystemDataset(
schema, None, ds.ParquetFileFormat(), pa.fs.LocalFileSystem(),
[str(basedir / "data_duplicate_columns.parquet")], [ds.ScalarExpression(True)])
```

then scanning fails:

```Java

>>> dataset.to_table()
...
ArrowInvalid: Multiple matches for FieldRef.Name(a) in a: int64
b: int64
a: int64
```

**Reporter**: [Joris Van den Bossche](https://issues.apache.org/jira/browse/ARROW-8210) / @jorisvandenbossche
#### Related issues:
- [[Python] Prevent corrupting files with Multiple matches for FieldRef.Name](https://github.com/apache/arrow/issues/32660) (is related to)
- [[C++][Dataset] Ensure that dataset code is robust to schemas with duplicate field names](https://github.com/apache/arrow/issues/23685) (is related to)

**Note**: *This issue was originally created as [ARROW-8210](https://issues.apache.org/jira/browse/ARROW-8210). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Reproduce the failures through pyarrow.dataset.dataset, UnionDatasetFactory, FileSystemDataset, and dataset.to_table using the Python examples in the issue. Trace the C++ dataset factory and scanning paths behind these entry points, then add focused coverage for duplicate columns with equal schemas and for scans without projection. Done means factory creation and unprojected scanning succeed for the shown duplicate-column dataset.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.