[Python] Best practice for adding a column for `__filename` without adding a duplicate field?
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
### Describe the usage question you have. Please include as many useful details as possible.
#### my problem
i have a `Dataset` and want to add the `__filename` field when creating a `Table` from this dataset. I've tried a few things:
#### attempt 1: use `dataset.scanner`
```python
columns = dataset.schema.names + ["__filename"] # ["__last_in_fragment"]
scanner = dataset.scanner(columns=columns)
my_table = scanner.to_table()
```
but with this, i get a duplicate column error when i try to access it with duckdb:
```python
con = duckdb.connect()
con.execute("SELECT * FROM my_table limit 5").fetch_df()
```
```
duckdb.InvalidInputException: Invalid Input Error: Attempting to execute an unsuccessful or closed pending query result
Error: Invalid Error: ArrowInvalid: Multiple matches for FieldRef.Name(__filename)
```
#### attempt 2: use a dict of columns
i have also tried creating the new column by specifying columns as expressions. but, `ds.field("__filename")` is not recognized:
```python
columns = {x: ds.field(x) for x in dataset.schema.names}
columns["filename"] = ds.field("__filename")
scanner = dataset.scanner(columns=columns)
```
```
Traceback (most recent call last)
----> scanner = dataset.scanner(columns=columns)
/opt/conda/envs/myenv/lib/python3.11/site-packages/pyarrow/_dataset.pyx:336, in pyarrow._dataset.Dataset.scanner()
/opt/conda/envs/myenv/lib/python3.11/site-packages/pyarrow/_dataset.pyx:2576, in pyarrow._dataset.Scanner.from_dataset()
/opt/conda/envs/myenv/lib/python3.11/site-packages/pyarrow/_dataset.pyx:2484, in pyarrow._dataset.Scanner._make_scan_options()
/opt/conda/envs/myenv/lib/python3.11/site-packages/pyarrow/_dataset.pyx:2381, in pyarrow._dataset._populate_builder()
/opt/conda/envs/myenv/lib/python3.11/site-packages/pyarrow/error.pxi:100, in pyarrow.lib.check_status()
ArrowInvalid: No match for FieldRef.Name(__filename) in
```
#### tl;dr
what is the best practice for adding fields like `__filename` to a table while avoiding duplicate column errors?
#### related issues
this might be related to
- https://github.com/apache/arrow/issues/24407
- https://github.com/dask/dask/issues/9251
#### versions
```
Linux-4.19.0-23-cloud-amd64-x86_64-with-glibc2.28
Python 3.11.0 | packaged by conda-forge (main, Jan 14 2023, 12:27:40) [GCC 11.3.0]
pyarrow 11.0.0
```
### Component(s)
Python
Contributor guide
Research direction
Start by reproducing the Python Dataset.scanner examples with pyarrow 11.0.0, including the columns list and expression dictionary using __filename. Read the scanner behavior around virtual fields and compare the resulting table with the reported DuckDB duplicate-field error. Done means the supported approach, or the required implementation scope, is clearly established and covered by an appropriate regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100