apache / apache/arrow

[Python] Best practice for adding a column for `__filename` without adding a duplicate field?

Open
#34,554 8 comments 3 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

### Describe the usage question you have. Please include as many useful details as possible.

#### my problem
i have a `Dataset` and want to add the `__filename` field when creating a `Table` from this dataset. I've tried a few things:

#### attempt 1: use `dataset.scanner`

```python
columns = dataset.schema.names + ["__filename"] # ["__last_in_fragment"]
scanner = dataset.scanner(columns=columns)
my_table = scanner.to_table()
```

but with this, i get a duplicate column error when i try to access it with duckdb:
```python
con = duckdb.connect()
con.execute("SELECT * FROM my_table limit 5").fetch_df()
```
```
duckdb.InvalidInputException: Invalid Input Error: Attempting to execute an unsuccessful or closed pending query result
Error: Invalid Error: ArrowInvalid: Multiple matches for FieldRef.Name(__filename)
```

#### attempt 2: use a dict of columns
i have also tried creating the new column by specifying columns as expressions. but, `ds.field("__filename")` is not recognized:

```python
columns = {x: ds.field(x) for x in dataset.schema.names}
columns["filename"] = ds.field("__filename")
scanner = dataset.scanner(columns=columns)
```

```
Traceback (most recent call last)
----> scanner = dataset.scanner(columns=columns)
/opt/conda/envs/myenv/lib/python3.11/site-packages/pyarrow/_dataset.pyx:336, in pyarrow._dataset.Dataset.scanner()
/opt/conda/envs/myenv/lib/python3.11/site-packages/pyarrow/_dataset.pyx:2576, in pyarrow._dataset.Scanner.from_dataset()
/opt/conda/envs/myenv/lib/python3.11/site-packages/pyarrow/_dataset.pyx:2484, in pyarrow._dataset.Scanner._make_scan_options()
/opt/conda/envs/myenv/lib/python3.11/site-packages/pyarrow/_dataset.pyx:2381, in pyarrow._dataset._populate_builder()
/opt/conda/envs/myenv/lib/python3.11/site-packages/pyarrow/error.pxi:100, in pyarrow.lib.check_status()
ArrowInvalid: No match for FieldRef.Name(__filename) in
```

#### tl;dr
what is the best practice for adding fields like `__filename` to a table while avoiding duplicate column errors?

#### related issues

this might be related to
- https://github.com/apache/arrow/issues/24407
- https://github.com/dask/dask/issues/9251

#### versions

```
Linux-4.19.0-23-cloud-amd64-x86_64-with-glibc2.28
Python 3.11.0 | packaged by conda-forge (main, Jan 14 2023, 12:27:40) [GCC 11.3.0]
pyarrow 11.0.0
```

### Component(s)

Python

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the Python Dataset.scanner examples with pyarrow 11.0.0, including the columns list and expression dictionary using __filename. Read the scanner behavior around virtual fields and compare the resulting table with the reported DuckDB duplicate-field error. Done means the supported approach, or the required implementation scope, is clearly established and covered by an appropriate regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.