apache / apache/paimon

[Bug] timestamp reading from paimon is different from hive after hive-cloning

Open
#5,722 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

Paimon's version: 1.2-snapshot(2025/06/09)

### Compute Engine

Hive's version: hive 2.3.10

Flink's version: 1.20.1

### Minimal reproduce step

When selecting the paimon table which was cloned from hive through hive-clone action, the selected result for timestamp is different between paimon and hive.

**Note: file format is parquet.**

Minimal reproduce step:
1. create a table in hive and insert some data:
```sql
-- create a table
CREATE TABLE test.hive_test_table(
`a` int COMMENT 'The a field',
`ts` timestamp COMMENT 'The ts field'
) STORED AS PARQUET;

-- insert a value
INSERT INTO test.hive_test_table VALUES (1, '2025-06-03 16:00:00');

-- select
SELECT * FROM test.hive_test_table;
-- result: 1, 2025-06-03T16:00
```

2. run hive-clone action

3. select the cloned paimon table in flink
```sql
SELECT * FROM paimon_hive_test_table;
-- result: 1, 2025-06-03T08:00
```

The reaseon may be:
Hive will transform the timestamp to UTC for storage, and transform the stored timestamp to `writer.time.zone` or system default time zone when reading. While paimon will directly read the original timestamp stored in parquet file without any transforms, this leads to difference.

### What doesn't meet your expectations?

The selected result for timestamp has changed after cloning to paimon.

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Run the listed Hive SQL setup, hive-clone action, and Flink query using the stated versions and Parquet format. Compare the timestamp before and after cloning, then trace the clone and timestamp-reading paths to determine where timezone handling diverges. Done means the cloned Paimon query returns the same timestamp as Hive.

Written by the indexing model from the issue text.

Assessment

Tech stack
sql
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.