[EPIC] A collection of support for metadata columns in ListingTable
- Vorherrschende Sprache
- Rust
- Sterne
- 9.3k
- Forks
- 2.4k
- Ø Merge
- 3 T. 11 Std.
- Gemergte PRs (30 T.)
- 360
Beschreibung
Other systems support "metadata" columns when querying datasources. These metadata columns do not exist in the underlying data source but instead are related to the source
Common examples:
* Row number
* File name
* Row Group Number (parquet)
## DataBricks / Spark
It appears DataBricks / spark represents this concept as a struct column `_metadata` column with multiple fields
https://docs.databricks.com/aws/en/ingestion/file-metadata-column
```sql
SELECT
*
,_metadata
,_metadata.file_path
,_metadata.file_name
,_metadata.file_modification_time
FROM
json.`/path/to/table/data`
```
It looks like maybe spark/databricks used to support the `input_file_name()` function, but has moved to `_metadata`: https://pawankumarshukla1979.medium.com/tips-use-metadata-instead-of-input-file-name-function-in-databricks-runtime-10-5-and-above-b32766b0296b
## DuckDB
DuckDB seems to model this as additional parameters to the `read_parquet` function, specifically `file_name` and `file_row_number`:
https://duckdb.org/docs/stable/data/parquet/overview#parameters
Per @adriangb [last year](https://github.com/apache/datafusion/issues/15173#issuecomment-2858707706):
```
D select filename, sum(row_count) as row_count from read_parquet('/Users/adriangb/Downloads/data2/**/*_stats.parquet', filename=true) group by filename order by row_count desc limit 10;
Binder Error:
Option filename adds column "filename", but a column with this name is also in the file. Try setting a different name: filename=''
```
Related tickets for adding similar metadata features to DataFusion:
- [ ] https://github.com/apache/datafusion/issues/13975
- [ ] https://github.com/apache/datafusion/issues/15173
- [x] https://github.com/apache/datafusion/issues/6051
- [ ] https://github.com/apache/datafusion/issues/20132
- [x] https://github.com/apache/datafusion/issues/13261
- [ ] https://github.com/apache/datafusion/issues/18482
Beitragsleitfaden
Rechercherichtung
Es werden keine Quelldateien, Tests oder Einstiegspunkte für die Implementierung genannt. Beginne damit, die verknüpften DataFusion-Tickets sowie die Metadatenbeispiele von Spark und DuckDB zu prüfen, und kläre anschließend den gewünschten Umfang der Metadatenspalten und die Akzeptanzkriterien, bevor du Implementierungsarbeiten identifizierst.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Bewertung
- Tech-Stack
- rust, sql
- Bereich
- backend-api-design, data-engineering
- Issue-Typ
- Feature
- Schwierigkeit
- 5/5
- Geschätzter Aufwand
- Über eine Woche
- Aktivitätsstatus
- Ruhig
- Klarheit
- Muss geklärt werden
- Anfängerfreundlichkeit
- 25/100