apache / apache/hudi

[SUPPORT] Hudi Timestamp Timezone is not set from Flink

Open
#7,217 1 comment 0 reactions 0 assignees View on GitHub
engine:flink priority:medium
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**_Tips before filing an issue_**

- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?

- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.

- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.

**Describe the problem you faced**

Compaction is not working for Hudi 0.12.1 and Flink 1.15.2 with the following configurations:
`'hoodie.compact.inline': 'true',
'hoodie.compact.inline.max.delta.commits':3,
'hoodie.table.timeline.timezone':'utc'`

We see a conflict between timeline timezones as below
compactionInstant (20221115171352311) Instants :[[20221115224210227__deltacommit__COMPLETED]
Hudi Cli:
`Error: Failed to run compaction for 20221115171043695

java.lang.IllegalArgumentException: Following instants have timestamps >= compactionInstant (20221115171352311) Instants :[[20221115224210227__deltacommit__COMPLETED], [20221115224244465__deltacommit__COMPLETED], [20221115224253081__deltacommit__COMPLETED], [20221115224303021__deltacommit__COMPLETED]]`

Flink logs:
`[compact_plan_generate (1/1)#2] INFO org.apache.flink.streaming.api.operators.AbstractStreamOperator - No compaction plan for checkpoint 8`

**To Reproduce**

Steps to reproduce the behavior:

1. Write to hudi with following config
`'table.type' = 'MERGE_ON_READ'
'hoodie.compact.inline': 'true',
'hoodie.compact.inline.max.delta.commits':3,
'hoodie.table.timeline.timezone':'utc'`

**Expected behavior**

The compaction should be scheduled as per the timezone specified in config options.

**Environment Description**

* Hudi version : 0.12.1

* Flink version : 1.15.2

* Hive version :

* Hadoop version :

* Storage (HDFS/S3/GCS..) :

* Running on Docker? (yes/no) : no

**Additional context**

Add any other context about the problem here.

**Stacktrace**

`[compact_plan_generate (1/1)#2] INFO org.apache.flink.streaming.api.operators.AbstractStreamOperator - No compaction plan for checkpoint 8`

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the Flink 1.15.2 compaction scenario with Hudi 0.12.1, MERGE_ON_READ, inline compaction, and the UTC timeline timezone setting. Compare the compaction instant with the delta-commit timestamps shown in the Hudi CLI error and Flink log. Done means compaction is scheduled according to the configured timezone without the timestamp conflict.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.