[EPIC] A collection of support for metadata columns in ListingTable
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
Other systems support "metadata" columns when querying datasources. These metadata columns do not exist in the underlying data source but instead are related to the source
Common examples:
* Row number
* File name
* Row Group Number (parquet)
## DataBricks / Spark
It appears DataBricks / spark represents this concept as a struct column `_metadata` column with multiple fields
https://docs.databricks.com/aws/en/ingestion/file-metadata-column
```sql
SELECT
*
,_metadata
,_metadata.file_path
,_metadata.file_name
,_metadata.file_modification_time
FROM
json.`/path/to/table/data`
```
It looks like maybe spark/databricks used to support the `input_file_name()` function, but has moved to `_metadata`: https://pawankumarshukla1979.medium.com/tips-use-metadata-instead-of-input-file-name-function-in-databricks-runtime-10-5-and-above-b32766b0296b
## DuckDB
DuckDB seems to model this as additional parameters to the `read_parquet` function, specifically `file_name` and `file_row_number`:
https://duckdb.org/docs/stable/data/parquet/overview#parameters
Per @adriangb [last year](https://github.com/apache/datafusion/issues/15173#issuecomment-2858707706):
```
D select filename, sum(row_count) as row_count from read_parquet('/Users/adriangb/Downloads/data2/**/*_stats.parquet', filename=true) group by filename order by row_count desc limit 10;
Binder Error:
Option filename adds column "filename", but a column with this name is also in the file. Try setting a different name: filename=''
```
Related tickets for adding similar metadata features to DataFusion:
- [ ] https://github.com/apache/datafusion/issues/13975
- [ ] https://github.com/apache/datafusion/issues/15173
- [x] https://github.com/apache/datafusion/issues/6051
- [ ] https://github.com/apache/datafusion/issues/20132
- [x] https://github.com/apache/datafusion/issues/13261
- [ ] https://github.com/apache/datafusion/issues/18482
Contributor guide
Research direction
No source files, tests, or implementation entry points are mentioned. Start by reviewing the linked DataFusion tickets and the Spark and DuckDB metadata examples, then clarify the desired metadata-column scope and acceptance criteria before identifying implementation work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- backend-api-design, data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100