refactor: unify ColumnSource::Add and AddStructConstant using Literal
- Dominant language
- Rust
- Stars
- 1.4k
- Forks
- 567
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 93
Description
follow up from #2668
The `ColumnSource` enum in `RecordBatchTransformer` currently has two separate variants for constant columns:
```rust
Add {
target_type: DataType,
value: Option,
},
AddStructConstant {
fields: Fields,
child_values: Vec>,
},
```
`AddStructConstant` was added in #2668 for the `_partition` metadata column because Add only supports primitive values. However, iceberg-rust already has Literal::Struct in crates/iceberg/src/spec/values/literal.rs which can represent struct values natively.
### Proposed refactoring
1. Change ColumnSource::Add to use Option instead of Option:
Add {
target_type: DataType,
value: Option,
},
2. Remove ColumnSource::AddStructConstant — struct constants become a Literal::Struct inside Add.
3. Update create_column (and create_struct_column) to handle Literal::Struct by recursively constructing child arrays, similar to how AddStructConstant works today.
4. This also opens the door for constant_fields: HashMap to eventually support struct-valued constants (e.g., if Datum gains a struct variant), which would allow the _partition column to use the same constant_fields path as _file and identity-partitioned columns.
### Benefits
- Eliminates a special-case enum variant
- Aligns the constant-column infrastructure with iceberg-rust's existing Literal type system
- Makes future metadata columns or complex-typed constants straightforward to add
Contributor guide
Research direction
Start by locating RecordBatchTransformer and its ColumnSource enum, then read crates/iceberg/src/spec/values/literal.rs for Literal::Struct. Trace create_column and create_struct_column, using the existing AddStructConstant behavior as the reference for recursive child-array construction. Done means Add accepts Literal, AddStructConstant is removed, and struct constants continue to produce equivalent columns.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100