[SUPPORT] Hudi Timestamp Timezone is not set from Flink
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**_Tips before filing an issue_**
- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
**Describe the problem you faced**
Compaction is not working for Hudi 0.12.1 and Flink 1.15.2 with the following configurations:
`'hoodie.compact.inline': 'true',
'hoodie.compact.inline.max.delta.commits':3,
'hoodie.table.timeline.timezone':'utc'`
We see a conflict between timeline timezones as below
compactionInstant (20221115171352311) Instants :[[20221115224210227__deltacommit__COMPLETED]
Hudi Cli:
`Error: Failed to run compaction for 20221115171043695
java.lang.IllegalArgumentException: Following instants have timestamps >= compactionInstant (20221115171352311) Instants :[[20221115224210227__deltacommit__COMPLETED], [20221115224244465__deltacommit__COMPLETED], [20221115224253081__deltacommit__COMPLETED], [20221115224303021__deltacommit__COMPLETED]]`
Flink logs:
`[compact_plan_generate (1/1)#2] INFO org.apache.flink.streaming.api.operators.AbstractStreamOperator - No compaction plan for checkpoint 8`
**To Reproduce**
Steps to reproduce the behavior:
1. Write to hudi with following config
`'table.type' = 'MERGE_ON_READ'
'hoodie.compact.inline': 'true',
'hoodie.compact.inline.max.delta.commits':3,
'hoodie.table.timeline.timezone':'utc'`
**Expected behavior**
The compaction should be scheduled as per the timezone specified in config options.
**Environment Description**
* Hudi version : 0.12.1
* Flink version : 1.15.2
* Hive version :
* Hadoop version :
* Storage (HDFS/S3/GCS..) :
* Running on Docker? (yes/no) : no
**Additional context**
Add any other context about the problem here.
**Stacktrace**
`[compact_plan_generate (1/1)#2] INFO org.apache.flink.streaming.api.operators.AbstractStreamOperator - No compaction plan for checkpoint 8`
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the Flink 1.15.2 compaction scenario with Hudi 0.12.1, MERGE_ON_READ, inline compaction, and the UTC timeline timezone setting. Compare the compaction instant with the delta-commit timestamps shown in the Hudi CLI error and Flink log. Done means compaction is scheduled according to the configured timezone without the timestamp conflict.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100