[Bug] The Hive connector encounters time offset issues when reading and writing data of the timestamp with local time zone field type.
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Paimon version
paimon-1.0.1
### Compute Engine
hive-3.1.2
### Minimal reproduce step
step 1: create a table and insert a data using hivesql
``` sql
-- hivesql
CREATE TABLE test.hive_test_table(
`a` int COMMENT 'The a field',
`ts` timestamp with local time zone COMMENT 'The ts field'
)
STORED BY 'org.apache.paimon.hive.PaimonStorageHandler';
-- insert a row
insert into test.hive_test_table values (1, '2025-05-06 19:00:00');
-- select it
hive> select * from test.hive_test_table;
OK
1 2025-05-06 19:00:00.0 PRC
```
It looks correct.
step2: use sparksql to query the data
``` sql
spark-sql (test)> select * from test.hive_test_table;
1 2025-05-07 03:00:00
```
The result goes wrong.
step3: use sparksql to insert a data
``` sql
insert into test.hive_test_table values (2, '2025-05-06 19:02:00');
```
step4: use hivesql/saprksql/flinksql to query all data
``` sql
-- hivesql
hive> select * from test.hive_test_table;
OK
1 2025-05-06 19:00:00.0 PRC
2 2025-05-06 11:02:00.0 PRC
-- sparksql
spark-sql (test)> select * from test.hive_test_table;
1 2025-05-07 03:00:00
2 2025-05-06 19:02:00
-- flinksql
Flink SQL> select * from test.hive_test_table;
+---+----------------------------+
| a | ts |
+---+----------------------------+
| 1 | 2025-05-07 03:00:00.000000 |
| 2 | 2025-05-06 19:02:00.000000 |
+---+----------------------------+
```
### What doesn't meet your expectations?
The Hive connector exhibits inconsistent behavior compared to two other engines when reading and writing data of the timestamp with local time zone field type.
### Anything else?
_No response_
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the timestamp-with-local-time-zone case through the Hive connector using Hive SQL, Spark SQL, and Flink SQL, following the CREATE TABLE and insert examples in the issue. Compare the values returned after each engine reads and writes the table, and trace the PaimonStorageHandler integration first. Done means all three engines preserve the intended timestamp and time-zone representation consistently.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark, sql
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100