apache / apache/hudi

Follow up all timeline related apis and methods to honor timezone if overridden

Open
#15,907 1 comment 0 reactions 0 assignees View on GitHub
area:reader area:table-service area:writer component:archiving from-jira priority:high type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

I see that HoodieTableConfig has a config to let users to override timezone for commit time generation. but looks like there are some places where we use current system's zone instead of honoring the config. 
{code:java}
public static Date parseDateFromInstantTime(String timestamp) throws ParseException {
try {
// Enables backwards compatibility with non-millisecond granularity instants
String timestampInMillis = timestamp;
if (isSecondGranularity(timestamp)) {
// Add milliseconds to the instant in order to parse successfully
timestampInMillis = timestamp + DEFAULT_MILLIS_EXT;
} else if (timestamp.length() > MILLIS_INSTANT_TIMESTAMP_FORMAT_LENGTH) {
// compaction and cleaning in metadata has special format. handling it by trimming extra chars and treating it with ms granularity
timestampInMillis = timestamp.substring(0, MILLIS_INSTANT_TIMESTAMP_FORMAT_LENGTH);
}

LocalDateTime dt = LocalDateTime.parse(timestampInMillis, MILLIS_INSTANT_TIME_FORMATTER);
return Date.from(dt.atZone(ZoneId.systemDefault()).toInstant());
} catch (DateTimeParseException e) {
throw new ParseException(e.getMessage(), e.getErrorIndex());
}
} {code}
 

also, we are adding a guard rail to archival that archival should not go past clean commits. when clean is based on hours, we have a fix which may not work for other time zones. Ref patch: https://github.com/apache/hudi/pull/8422

 

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-6097
- Type: Improvement

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue points to parseDateFromInstantTime and HoodieTableConfig, plus archival clean-commit handling; start by tracing those timeline APIs and comparing the timezone override with the system-default conversion. Use PR #8422 as context, then verify every affected path honors the configured timezone, including hour-based cleaning and archival.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.