Physical-plan serialization drops ProjectionExec metadata
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Physical-plan serialization
Before `ProjectionExecNode` serialized:
```protobuf
PhysicalPlanNode input = 1;
repeated PhysicalExprNode expr = 2;
repeated string expr_name = 3;
```
On decode, DataFusion reconstructed the projection which is fine when the complete output schema can be derived from the expression and input but metadata cannot be.
Given:
```text
Before serialization:
Input field:
value: Int32
metadata: {}
Projection expression:
value@0 AS value
ProjectionExec output schema:
value: Int32
field metadata: {"field-key": "field-value"}
schema metadata: {"schema-key": "schema-value"}
---
After serialization and decoding:
ProjectionExec output schema:
value: Int32
field metadata: {}
schema metadata: {}
```
---
this was roginally found in #24670 but not fixed there since it was out of optimizer scope / unrelated bug
Contributor guide
Research direction
Start at the physical-plan serialization and decoding path for ProjectionExecNode, using the before-and-after example in the issue to reproduce the metadata loss. Trace how the projection output schema is reconstructed, then verify that field metadata and schema metadata survive serialization and decoding.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100