apache / apache/datafusion

Document Schema metadata expectations

Open
#12,736 10 comments 0 reactions 1 assignee Assigned to @alamb View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

There is an (implicit) assumption that metadata attached to Schema is preserved during certain operations in DataFusion.

However, this expectation is clearly not well tested or documented (e.g. see https://github.com/apache/datafusion/issues/12733)

### Describe the solution you'd like

I would like the assumptions documented

### Describe alternatives you've considered

I suggest documentation on in https://docs.rs/datafusion/latest/datafusion/logical_expr/enum.LogicalPlan.html that explains the high level assumptions

Then add a note /link to that section from the optimizers:
https://docs.rs/datafusion/latest/datafusion/optimizer/trait.AnalyzerRule.html
https://docs.rs/datafusion/latest/datafusion/optimizer/trait.OptimizerRule.html
https://docs.rs/datafusion/latest/datafusion/physical_optimizer/trait.PhysicalOptimizerRule.html

My understanding of the high level assumptions are:
* schema level metadata: always passed through
* field level metadata: when there is a clear 1-1 correspondence from an input column with metadata to an output column, the metadata should be preserved

Examples
* `PROJECT(a, b+c)` --> field metadata on`a` should be preserved, no field metadata on `b+c`
* `SUM(a) .. GROUP BY b` --> field metadata on `b` is preserved, not on `a`

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.