PartitionExpr cannot be reconstructed from its public API
- Dominant language
- Rust
- Stars
- 1.4k
- Forks
- 567
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 93
Description
### Is your feature request related to a problem or challenge?
A distributed query engine plans on one node and executes on others, so physical
plans must survive serialization. `project_with_partition` injects a
`PartitionExpr` into the projection for partitioned writes, and that expression
cannot make the trip.
It wraps a `PartitionValueCalculator`, which is a live non-serializable state. The two
things that are serializable (the `PartitionSpec` and the table schema) are
consumed at construction and not retained, so nothing on the expression says
what produced it.
The spec alone is not enough to rebuild the calculator: `PartitionSpec` stores
only `spec_id` and `fields`, referring to columns by `source_id`. The schema is
what resolves those ids to real columns and determines the partition type.
Reaching for the schema elsewhere at decode time isn't possible either:
`PhysicalExtensionCodec::try_encode_expr` / `try_decode_expr` receive only the
expression, with no tab
### Describe the solution you'd like
Retain both inputs on `PartitionExpr` and expose them:
- `try_new(partition_spec, table_schema)` replacing the private
`new(calculator, spec)`, constructing the calculator internally so the
retained inputs and the calculator cannot drift apart.
- `partition_spec()` and `table_schema()` accessors.
Additive: `project_with_partition` keeps its signature, and `new` was private,
so no existing caller changes.
### Willingness to contribute
I can contribute to this feature independently
Contributor guide
Research direction
Start by locating PartitionExpr, its private new(calculator, spec), and project_with_partition, then read PhysicalExtensionCodec::try_encode_expr and try_decode_expr to understand the serialization boundary. Retain the PartitionSpec and table schema through try_new and expose both accessors while keeping project_with_partition unchanged; done means PartitionExpr has enough public information to be reconstructed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100