apache / apache/iceberg-rust

Year and Month transforms depend on the input array's timezone tag; Iceberg computes them in UTC

Open
#3,142 0 comments 2 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 2h
Merged PRs (30d)
93

Description

### Apache Iceberg Rust version

main @ `3d84c81353b1b23b6e4ae8eea8f8a021cc6927a7`

### Describe the bug

`Year::transform` and `Month::transform` in `crates/iceberg/src/transform/temporal.rs` extract the calendar field with Arrow's `date_part`:

```rust
impl TransformFunction for Year {
fn transform(&self, input: ArrayRef) -> Result {
let array = date_part(&input, DatePart::Year)?;
...unary(|v| v - UNIX_EPOCH_YEAR)
}
}
```

`date_part` is timezone-aware: for a `Timestamp(Microsecond, Some(tz))` array it converts each value into `tz` before reading the year. Iceberg computes all four temporal transforms in UTC — `DateTimeUtil.convertMicros` builds an `OffsetDateTime` at `ZoneOffset.UTC` regardless of anything else. So for an array carrying a non-UTC tag the two disagree.

`Day` and `Hour` are not affected — they do floor division on the raw epoch value.

**This is also internally inconsistent.** `transform_literal` on the same two structs does the UTC thing:

```rust
(PrimitiveType::Timestamptz, PrimitiveLiteral::Long(v)) => Self::timestamp_to_year_micros(*v)?
// -> DateTime::from_timestamp_micros(v).year() - UNIX_EPOCH_YEAR
```

`DateTime::from_timestamp_micros` is UTC, so `transform` and `transform_literal` return different values for the same instant when the array is tagged non-UTC. Predicate projection and partition-value computation would then disagree with each other.

In practice iceberg-rust's own callers pass arrays built from an Iceberg schema, where `Timestamptz` is tagged `+00:00`, so this is latent rather than actively broken inside the crate. It is reachable for any external caller handing the transform an Arrow array it built itself.

### To Reproduce

```rust
let tagged: ArrayRef = Arc::new(
TimestampMicrosecondArray::from(vec![-1i64]).with_timezone("Asia/Kathmandu"),
);
let years = create_transform_function(&Transform::Year).unwrap().transform(tagged).unwrap();
// yields 0; Iceberg Java's DateTimeUtil.microsToYears(-1) is -1
```

`-1` micros is `1969-12-31T23:59:59.999999Z`, which is `1970-01-01T05:44:59.999999` in Kathmandu, so `date_part` reads year 1970 and the transform returns 0 where Iceberg returns -1. Any tag with a non-zero offset produces a similar disagreement for values near a year or month boundary.

### Expected behavior

`Year` and `Month` computed in UTC for every input, matching `DateTimeUtil` and matching their own `transform_literal`. Computing from the epoch value directly — the way `Day` and `Hour` already do — gives that. Normalising the array to UTC before calling `date_part` would also work but keeps a dependency on the tag being present and correct.

Two smaller things in the same area, if they are worth folding in:

- `Year::transform` / `Month::transform` inherit `chrono`'s date range through `date_part`, which stops at about year 262143, while Java's `LocalDate` covers the whole `i32` epoch-day domain (`i32::MAX` days is +5881580-07-11). Epoch days beyond that are representable in an Arrow `Date32` but not handled here.
- `Year::transform` has no `Date32` arm distinct from the timestamp ones; it relies on `date_part` accepting both. Fine today, just worth noting if the kernel is rewritten.

### Willingness to contribute

I would be willing to contribute a fix for this bug with guidance from the Iceberg community.

---

**How this was found.** Apache DataFusion Comet is adding native kernels for Iceberg's Spark system functions (apache/datafusion-comet#5638). We cross-check those kernels against `create_transform_function` over boundary inputs so a future iceberg-rust bump cannot silently desynchronise a partitioned write. The timezone-tag dependency is the reason `years` and `months` could not be delegated to iceberg-rust while `bucket`, `days`, and `hours` could.

_This report was drafted with LLM assistance (Claude Code); the behaviour was verified against Iceberg Java on a JVM._

Contributor guide

Open the contributing guide

Research direction

Start in crates/iceberg/src/transform/temporal.rs and inspect Year::transform, Month::transform, and their transform_literal implementations. Reproduce the shown non-UTC TimestampMicrosecondArray case, then verify that year and month transforms use UTC consistently and match the literal results, with regression coverage for the timezone-tagged boundary.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.