apache / apache/hudi

Timeline layout v2 path configs are persisted into hoodie.properties of tables below version 8

Open
#19,562 0 comments 0 reactions 1 assignee Claimed by @yihua View on GitHub
area:config area:core priority:medium
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

A table created below table version 8 gets `hoodie.timeline.path=timeline` and `hoodie.timeline.history.path=history` written into its `hoodie.properties`, even though it does not use the layout those configs describe.

Both configs were introduced with timeline layout version 2 in 1.0.0 and are read only by `TimelinePathProviderV2`, which is selected exclusively at layout version 2 (table version 8 and above). A table below that version keeps its timeline directly under `.hoodie` (`TimelinePathProviderV1` hardcodes it) and resolves its archived timeline through `hoodie.archivelog.folder`. Neither config has any meaning there.

The cause is that `HoodieTableConfig.dropInvalidConfigs()` only strips a config that declares a `sinceVersion`:

```java
if (!configProperty.getSinceVersion().isPresent()) {
return true; // never dropped
}
```

`RECORD_MERGE_MODE` declares `.sinceVersion("1.0.0")` and is correctly dropped from a version 6 table; `TIMELINE_PATH` and `TIMELINE_HISTORY_PATH` declare nothing and survive.

The immediate effect is cosmetic, since every consumer resolves these through `getStringOrDefault`. It matters because table version 6 is the interoperable format used when writing with a 1.x binary for a 0.x reader, and the 0.x line has no definition for either key. Recording storage layout that the table does not have is also a trap for anything that later starts honouring the values.

Reproduce by creating a table with `hoodie.write.table.version=6` and reading `hoodie.properties`.

Fix: declare the introducing version on both so the existing gate drops them below table version 8.

Note that `hoodie.table.format` looks similar but must not be handled this way: the table format SPI is orthogonal to the table version, nothing gates a custom format on table version 9, and no upgrade handler restores the config, so gating it on `sinceVersion` would permanently discard a custom format on any table below version 9.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.