apache / apache/datafusion

Physical-plan serialization drops ProjectionExec metadata

Open
#24,695 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Physical-plan serialization
Before `ProjectionExecNode` serialized:
```protobuf
PhysicalPlanNode input = 1;
repeated PhysicalExprNode expr = 2;
repeated string expr_name = 3;
```

On decode, DataFusion reconstructed the projection which is fine when the complete output schema can be derived from the expression and input but metadata cannot be.

Given:
```text
Before serialization:

Input field:
value: Int32
metadata: {}

Projection expression:
value@0 AS value

ProjectionExec output schema:
value: Int32
field metadata: {"field-key": "field-value"}
schema metadata: {"schema-key": "schema-value"}

---

After serialization and decoding:

ProjectionExec output schema:
value: Int32
field metadata: {}
schema metadata: {}
```

---

this was roginally found in #24670 but not fixed there since it was out of optimizer scope / unrelated bug

Contributor guide

Open the contributing guide

Research direction

Start at the physical-plan serialization and decoding path for ProjectionExecNode, using the before-and-after example in the issue to reproduce the metadata loss. Trace how the projection output schema is reconstructed, then verify that field metadata and schema metadata survive serialization and decoding.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.