Year and Month transforms depend on the input array's timezone tag; Iceberg computes them in UTC
- Dominant language
- Rust
- Stars
- 1.4k
- Forks
- 567
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 93
Description
### Apache Iceberg Rust version
main @ `3d84c81353b1b23b6e4ae8eea8f8a021cc6927a7`
### Describe the bug
`Year::transform` and `Month::transform` in `crates/iceberg/src/transform/temporal.rs` extract the calendar field with Arrow's `date_part`:
```rust
impl TransformFunction for Year {
fn transform(&self, input: ArrayRef) -> Result {
let array = date_part(&input, DatePart::Year)?;
...unary(|v| v - UNIX_EPOCH_YEAR)
}
}
```
`date_part` is timezone-aware: for a `Timestamp(Microsecond, Some(tz))` array it converts each value into `tz` before reading the year. Iceberg computes all four temporal transforms in UTC — `DateTimeUtil.convertMicros` builds an `OffsetDateTime` at `ZoneOffset.UTC` regardless of anything else. So for an array carrying a non-UTC tag the two disagree.
`Day` and `Hour` are not affected — they do floor division on the raw epoch value.
**This is also internally inconsistent.** `transform_literal` on the same two structs does the UTC thing:
```rust
(PrimitiveType::Timestamptz, PrimitiveLiteral::Long(v)) => Self::timestamp_to_year_micros(*v)?
// -> DateTime::from_timestamp_micros(v).year() - UNIX_EPOCH_YEAR
```
`DateTime::from_timestamp_micros` is UTC, so `transform` and `transform_literal` return different values for the same instant when the array is tagged non-UTC. Predicate projection and partition-value computation would then disagree with each other.
In practice iceberg-rust's own callers pass arrays built from an Iceberg schema, where `Timestamptz` is tagged `+00:00`, so this is latent rather than actively broken inside the crate. It is reachable for any external caller handing the transform an Arrow array it built itself.
### To Reproduce
```rust
let tagged: ArrayRef = Arc::new(
TimestampMicrosecondArray::from(vec![-1i64]).with_timezone("Asia/Kathmandu"),
);
let years = create_transform_function(&Transform::Year).unwrap().transform(tagged).unwrap();
// yields 0; Iceberg Java's DateTimeUtil.microsToYears(-1) is -1
```
`-1` micros is `1969-12-31T23:59:59.999999Z`, which is `1970-01-01T05:44:59.999999` in Kathmandu, so `date_part` reads year 1970 and the transform returns 0 where Iceberg returns -1. Any tag with a non-zero offset produces a similar disagreement for values near a year or month boundary.
### Expected behavior
`Year` and `Month` computed in UTC for every input, matching `DateTimeUtil` and matching their own `transform_literal`. Computing from the epoch value directly — the way `Day` and `Hour` already do — gives that. Normalising the array to UTC before calling `date_part` would also work but keeps a dependency on the tag being present and correct.
Two smaller things in the same area, if they are worth folding in:
- `Year::transform` / `Month::transform` inherit `chrono`'s date range through `date_part`, which stops at about year 262143, while Java's `LocalDate` covers the whole `i32` epoch-day domain (`i32::MAX` days is +5881580-07-11). Epoch days beyond that are representable in an Arrow `Date32` but not handled here.
- `Year::transform` has no `Date32` arm distinct from the timestamp ones; it relies on `date_part` accepting both. Fine today, just worth noting if the kernel is rewritten.
### Willingness to contribute
I would be willing to contribute a fix for this bug with guidance from the Iceberg community.
---
**How this was found.** Apache DataFusion Comet is adding native kernels for Iceberg's Spark system functions (apache/datafusion-comet#5638). We cross-check those kernels against `create_transform_function` over boundary inputs so a future iceberg-rust bump cannot silently desynchronise a partitioned write. The timezone-tag dependency is the reason `years` and `months` could not be delegated to iceberg-rust while `bucket`, `days`, and `hours` could.
_This report was drafted with LLM assistance (Claude Code); the behaviour was verified against Iceberg Java on a JVM._
Contributor guide
Research direction
Start in crates/iceberg/src/transform/temporal.rs and inspect Year::transform, Month::transform, and their transform_literal implementations. Reproduce the shown non-UTC TimestampMicrosecondArray case, then verify that year and month transforms use UTC consistently and match the literal results, with regression coverage for the timezone-tagged boundary.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100