apache / apache/paimon

[Bug] The Hive connector encounters time offset issues when reading and writing data of the timestamp with local time zone field type.

Open
#5,568 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

paimon-1.0.1

### Compute Engine

hive-3.1.2

### Minimal reproduce step

step 1: create a table and insert a data using hivesql
``` sql
-- hivesql
CREATE TABLE test.hive_test_table(
`a` int COMMENT 'The a field',
`ts` timestamp with local time zone COMMENT 'The ts field'
)
STORED BY 'org.apache.paimon.hive.PaimonStorageHandler';

-- insert a row
insert into test.hive_test_table values (1, '2025-05-06 19:00:00');

-- select it
hive> select * from test.hive_test_table;
OK
1 2025-05-06 19:00:00.0 PRC
```
It looks correct.

step2: use sparksql to query the data
``` sql
spark-sql (test)> select * from test.hive_test_table;
1 2025-05-07 03:00:00
```
The result goes wrong.

step3: use sparksql to insert a data
``` sql
insert into test.hive_test_table values (2, '2025-05-06 19:02:00');
```

step4: use hivesql/saprksql/flinksql to query all data
``` sql
-- hivesql
hive> select * from test.hive_test_table;
OK
1 2025-05-06 19:00:00.0 PRC
2 2025-05-06 11:02:00.0 PRC

-- sparksql
spark-sql (test)> select * from test.hive_test_table;
1 2025-05-07 03:00:00
2 2025-05-06 19:02:00

-- flinksql
Flink SQL> select * from test.hive_test_table;
+---+----------------------------+
| a | ts |
+---+----------------------------+
| 1 | 2025-05-07 03:00:00.000000 |
| 2 | 2025-05-06 19:02:00.000000 |
+---+----------------------------+
```

### What doesn't meet your expectations?

The Hive connector exhibits inconsistent behavior compared to two other engines when reading and writing data of the timestamp with local time zone field type.

### Anything else?

_No response_

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the timestamp-with-local-time-zone case through the Hive connector using Hive SQL, Spark SQL, and Flink SQL, following the CREATE TABLE and insert examples in the issue. Compare the values returned after each engine reads and writes the table, and trace the PaimonStorageHandler integration first. Done means all three engines preserve the intended timestamp and time-zone representation consistently.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark, sql
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.