apache / apache/iceberg-rust

refactor: unify ColumnSource::Add and AddStructConstant using Literal

Open
#2,699 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 2h
Merged PRs (30d)
93

Description

follow up from #2668

The `ColumnSource` enum in `RecordBatchTransformer` currently has two separate variants for constant columns:

```rust
Add {
target_type: DataType,
value: Option,
},
AddStructConstant {
fields: Fields,
child_values: Vec>,
},
```

`AddStructConstant` was added in #2668 for the `_partition` metadata column because Add only supports primitive values. However, iceberg-rust already has Literal::Struct in crates/iceberg/src/spec/values/literal.rs which can represent struct values natively.

### Proposed refactoring

1. Change ColumnSource::Add to use Option instead of Option:
Add {
target_type: DataType,
value: Option,
},
2. Remove ColumnSource::AddStructConstant — struct constants become a Literal::Struct inside Add.
3. Update create_column (and create_struct_column) to handle Literal::Struct by recursively constructing child arrays, similar to how AddStructConstant works today.
4. This also opens the door for constant_fields: HashMap to eventually support struct-valued constants (e.g., if Datum gains a struct variant), which would allow the _partition column to use the same constant_fields path as _file and identity-partitioned columns.

### Benefits

- Eliminates a special-case enum variant
- Aligns the constant-column infrastructure with iceberg-rust's existing Literal type system
- Makes future metadata columns or complex-typed constants straightforward to add

Contributor guide

Open the contributing guide

Research direction

Start by locating RecordBatchTransformer and its ColumnSource enum, then read crates/iceberg/src/spec/values/literal.rs for Literal::Struct. Trace create_column and create_struct_column, using the existing AddStructConstant behavior as the reference for recursive child-array construction. Done means Add accepts Literal, AddStructConstant is removed, and struct constants continue to produce equivalent columns.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.