apache / apache/datafusion

[EPIC] A collection of support for metadata columns in ListingTable

Aperta
#20,135 30 commenti 3 reazioni 0 assegnatari Vedi su GitHub
PROPOSAL EPIC
Lingua principale
Rust
Stelle
9.3k
Fork
2.4k
Merge medio
3g 11h
PR unite (30g)
360

Descrizione

Other systems support "metadata" columns when querying datasources. These metadata columns do not exist in the underlying data source but instead are related to the source

Common examples:
* Row number
* File name
* Row Group Number (parquet)

## DataBricks / Spark

It appears DataBricks / spark represents this concept as a struct column `_metadata` column with multiple fields
https://docs.databricks.com/aws/en/ingestion/file-metadata-column

```sql
SELECT
*
,_metadata
,_metadata.file_path
,_metadata.file_name
,_metadata.file_modification_time
FROM
json.`/path/to/table/data`
```

It looks like maybe spark/databricks used to support the `input_file_name()` function, but has moved to `_metadata`: https://pawankumarshukla1979.medium.com/tips-use-metadata-instead-of-input-file-name-function-in-databricks-runtime-10-5-and-above-b32766b0296b

## DuckDB
DuckDB seems to model this as additional parameters to the `read_parquet` function, specifically `file_name` and `file_row_number`:
https://duckdb.org/docs/stable/data/parquet/overview#parameters

Per @adriangb [last year](https://github.com/apache/datafusion/issues/15173#issuecomment-2858707706):
```
D select filename, sum(row_count) as row_count from read_parquet('/Users/adriangb/Downloads/data2/**/*_stats.parquet', filename=true) group by filename order by row_count desc limit 10;
Binder Error:
Option filename adds column "filename", but a column with this name is also in the file. Try setting a different name: filename=''
```

Related tickets for adding similar metadata features to DataFusion:
- [ ] https://github.com/apache/datafusion/issues/13975
- [ ] https://github.com/apache/datafusion/issues/15173
- [x] https://github.com/apache/datafusion/issues/6051
- [ ] https://github.com/apache/datafusion/issues/20132
- [x] https://github.com/apache/datafusion/issues/13261
- [ ] https://github.com/apache/datafusion/issues/18482

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Non sono menzionati file sorgente, test o punti di ingresso dell’implementazione. Inizia esaminando i ticket DataFusion collegati e gli esempi di metadati di Spark e DuckDB, quindi chiarisci l’ambito desiderato delle colonne di metadati e i criteri di accettazione prima di individuare il lavoro di implementazione.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
rust, sql
Ambito
backend-api-design, data-engineering
Tipo di issue
Funzionalità
Difficoltà
5/5
Tempo stimato
Più di una settimana
Stato di attività
Tranquilla
Chiarezza
Da chiarire
Idoneità per principianti
25/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.