[C++][Dataset] Handling of duplicate columns in Dataset factory and scanning
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
While testing duplicate column names, I ran into multiple issues:
- Factory fails if there are duplicate columns, even for a single file
- In addition, we should also fix and/or test that factory works for duplicate columns if the schema's are equal
- Once a Dataset with duplicated columns is created, scanning without any column projection fails
—
My python reproducer:
```python
import pyarrow as pa
import pyarrow.parquet as pq
import pyarrow.dataset as ds
import pyarrow.fs
# create single parquet file with duplicated column names
table = pa.table([pa.array([1, 2, 3]), pa.array([4, 5, 6]), pa.array([7, 8, 9])], names=['a', 'b', 'a'])
pq.write_table(table, "data_duplicate_columns.parquet")
```
Factory fails:
```Java
dataset = ds.dataset("data_duplicate_columns.parquet", format="parquet")
...
~/scipy/repos/arrow/python/pyarrow/dataset.py in dataset(paths_or_factories, filesystem, partitioning, format)
346
347 factories = [_ensure_factory(f, **kwargs) for f in paths_or_factories]
--> 348 return UnionDatasetFactory(factories).finish()
349
350
ArrowInvalid: Can't unify schema with duplicate field names.
```
And when creating a Dataset manually:
```python
schema = pa.schema([('a', 'int64'), ('b', 'int64'), ('a', 'int64')])
dataset = ds.FileSystemDataset(
schema, None, ds.ParquetFileFormat(), pa.fs.LocalFileSystem(),
[str(basedir / "data_duplicate_columns.parquet")], [ds.ScalarExpression(True)])
```
then scanning fails:
```Java
>>> dataset.to_table()
...
ArrowInvalid: Multiple matches for FieldRef.Name(a) in a: int64
b: int64
a: int64
```
**Reporter**: [Joris Van den Bossche](https://issues.apache.org/jira/browse/ARROW-8210) / @jorisvandenbossche
#### Related issues:
- [[Python] Prevent corrupting files with Multiple matches for FieldRef.Name](https://github.com/apache/arrow/issues/32660) (is related to)
- [[C++][Dataset] Ensure that dataset code is robust to schemas with duplicate field names](https://github.com/apache/arrow/issues/23685) (is related to)
**Note**: *This issue was originally created as [ARROW-8210](https://issues.apache.org/jira/browse/ARROW-8210). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Reproduce the failures through pyarrow.dataset.dataset, UnionDatasetFactory, FileSystemDataset, and dataset.to_table using the Python examples in the issue. Trace the C++ dataset factory and scanning paths behind these entry points, then add focused coverage for duplicate columns with equal schemas and for scans without projection. Done means factory creation and unprojected scanning succeed for the shown duplicate-column dataset.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100