apache / apache/iceberg-rust

Writing UTC timestamp to an iceberg table fails

Open
#2,478 3 comments 1 reaction 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 2h
Merged PRs (30d)
93

Description

### Apache Iceberg Rust version

None

### Describe the bug

Writing UTC timestamp to an iceberg table fails in the arrow writer. [A discussion was had ](https://github.com/apache/arrow-rs/pull/9994) in arrow-rs if the logic should live there to slacken this and the suggestion was to handle this in application code similar to what datafusion does for this exact problem.

```
ArrowError(
"Incompatible type. Field 'timestamp' has type Timestamp(µs, \"+00:00\"), array has type Timestamp(µs, \"UTC\")",
),
```

### To Reproduce

_No response_

### Expected behavior

Don't fail, these two types are identical semantically

### Willingness to contribute

I can contribute a fix for this bug independently

Contributor guide

Open the contributing guide

Research direction

Start in the Iceberg arrow writer where the incompatible timestamp types are rejected, then review the linked arrow-rs discussion and DataFusion's handling of the same problem. Confirm the expected behavior by reproducing a write with UTC timestamps and verify that Timestamp(µs, "UTC") and Timestamp(µs, "+00:00") are accepted as semantically equivalent.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.