apache / apache/iceberg-rust

PartitionExpr cannot be reconstructed from its public API

Open
#3,002 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 2h
Merged PRs (30d)
93

Description

### Is your feature request related to a problem or challenge?

A distributed query engine plans on one node and executes on others, so physical
plans must survive serialization. `project_with_partition` injects a
`PartitionExpr` into the projection for partitioned writes, and that expression
cannot make the trip.

It wraps a `PartitionValueCalculator`, which is a live non-serializable state. The two
things that are serializable (the `PartitionSpec` and the table schema) are
consumed at construction and not retained, so nothing on the expression says
what produced it.

The spec alone is not enough to rebuild the calculator: `PartitionSpec` stores
only `spec_id` and `fields`, referring to columns by `source_id`. The schema is
what resolves those ids to real columns and determines the partition type.

Reaching for the schema elsewhere at decode time isn't possible either:
`PhysicalExtensionCodec::try_encode_expr` / `try_decode_expr` receive only the
expression, with no tab

### Describe the solution you'd like

Retain both inputs on `PartitionExpr` and expose them:

- `try_new(partition_spec, table_schema)` replacing the private
`new(calculator, spec)`, constructing the calculator internally so the
retained inputs and the calculator cannot drift apart.
- `partition_spec()` and `table_schema()` accessors.

Additive: `project_with_partition` keeps its signature, and `new` was private,
so no existing caller changes.

### Willingness to contribute

I can contribute to this feature independently

Contributor guide

Open the contributing guide

Research direction

Start by locating PartitionExpr, its private new(calculator, spec), and project_with_partition, then read PhysicalExtensionCodec::try_encode_expr and try_decode_expr to understand the serialization boundary. Retain the PartitionSpec and table schema through try_new and expose both accessors while keeping project_with_partition unchanged; done means PartitionExpr has enough public information to be reconstructed.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
distributed-systems
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.