Follow up all timeline related apis and methods to honor timezone if overridden
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
I see that HoodieTableConfig has a config to let users to override timezone for commit time generation. but looks like there are some places where we use current system's zone instead of honoring the config.
{code:java}
public static Date parseDateFromInstantTime(String timestamp) throws ParseException {
try {
// Enables backwards compatibility with non-millisecond granularity instants
String timestampInMillis = timestamp;
if (isSecondGranularity(timestamp)) {
// Add milliseconds to the instant in order to parse successfully
timestampInMillis = timestamp + DEFAULT_MILLIS_EXT;
} else if (timestamp.length() > MILLIS_INSTANT_TIMESTAMP_FORMAT_LENGTH) {
// compaction and cleaning in metadata has special format. handling it by trimming extra chars and treating it with ms granularity
timestampInMillis = timestamp.substring(0, MILLIS_INSTANT_TIMESTAMP_FORMAT_LENGTH);
}
LocalDateTime dt = LocalDateTime.parse(timestampInMillis, MILLIS_INSTANT_TIME_FORMATTER);
return Date.from(dt.atZone(ZoneId.systemDefault()).toInstant());
} catch (DateTimeParseException e) {
throw new ParseException(e.getMessage(), e.getErrorIndex());
}
} {code}
also, we are adding a guard rail to archival that archival should not go past clean commits. when clean is based on hours, we have a fix which may not work for other time zones. Ref patch: https://github.com/apache/hudi/pull/8422
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-6097
- Type: Improvement
Contributor guide
No contributing guide indexed for this repository
Research direction
The issue points to parseDateFromInstantTime and HoodieTableConfig, plus archival clean-commit handling; start by tracing those timeline APIs and comparing the timezone override with the system-default conversion. Use PR #8422 as context, then verify every affected path honors the configured timezone, including hour-based cleaning and archival.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100