apache / apache/hudi

Table creation writes the timeline layout v2 history folder into hoodie.archivelog.folder on tables below version 8

Open
#19,561 0 comments 0 reactions 1 assignee Claimed by @yihua View on GitHub
area:config area:core priority:medium
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

When a table is created at a write table version below 8, the table creation paths still write `hoodie.archivelog.folder=history` into `hoodie.properties`.

That value is the timeline layout version 2 default (`TIMELINE_HISTORY_PATH.defaultValue()`). A table below version 8 uses timeline layout version 1, where `TimelinePathProviderV1.getTimelineHistoryPath()` resolves the archived timeline through `hoodie.archivelog.folder`. So such a table archives into `.hoodie/history` rather than `.hoodie/archived`, which is what the 0.x line and `ARCHIVELOG_FOLDER.defaultValue()` produce everywhere else.

Reproduce by creating a table with `hoodie.write.table.version=6` through the streamer, the Spark SQL writer, either bootstrap executor, or the Flink `StreamerUtil`, then reading `hoodie.properties`.

The affected call sites hardcode the layout version 2 default with no table-version check, for example `StreamSync`:

```java
.setArchiveLogFolder(TIMELINE_HISTORY_PATH.defaultValue()) // "history", regardless of table version
```

Nothing is lost, since both the 0.x and 1.x lines read the persisted value. But the table diverges from every other table below version 8, and any tooling assuming the conventional `archived` directory (CLI, backup and restore scripts, manual inspection, external readers) looks in the wrong place. The metadata table already avoids this through a table-version-6 subclass that returns `ARCHIVELOG_FOLDER.defaultValue()`; the data table paths were missed.

There is also a latent interaction with downgrade: `EightToSevenDowngradeHandler` writes the converted archived timeline to a hardcoded `.hoodie/archived` without updating the config, so a version 8 table carrying `archivelog.folder=history` ends up with its archived timeline pointing at an empty directory after downgrade.

Fix: let the table builder supply `ARCHIVELOG_FOLDER.defaultValue()` and pass through only a user-configured value.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.