apache / apache/hudi

Table version 1 tables cannot be written to on the 0.x line: record key config conflict blocks the upgrade that would fix it

Open
#19,723 0 comments 0 reactions 1 assignee Claimed by @yihua View on GitHub
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Describe the problem you faced**

A table at table version 1 can never be written to again by the 0.x line. `HoodieWriterUtils.validateTableConfig` fails with a config conflict on the record key, and the upgrade that would resolve it never gets to run.

`hoodie.table.recordkey.fields` first shipped in 0.9.0 (HUDI-1659), so a table written at table version 1 does not carry it. The validation treats a non-empty write-config record key against an absent table-config record key as a conflict:

```
Config conflict(key current value existing value):
RecordKey: null
```

`OneToTwoUpgradeHandler` backfills `hoodie.table.recordkey.fields` from that same write config, so the table would heal itself on the next write. But validation runs before the upgrade does, so the write fails first. Through the DeltaStreamer this happens in the `StreamSyncService` constructor, before `BaseHoodieWriteClient.doInitTable -> tryUpgrade` is ever reached, and the stream dies at startup.

This was fixed on the 1.x line by HUDI-8795 (#12544), which skips the record-key check when the table version is 1. `branch-0.x` never received it, so the 0.x line still cannot upgrade these tables.

**To Reproduce**

1. Create a table whose `hoodie.properties` has `hoodie.table.version=1` and no `hoodie.table.recordkey.fields` (the shape a 0.8.x writer leaves behind).
2. Write to it with a write config that sets `hoodie.datasource.write.recordkey.field`.
3. The write fails with the config conflict above, whether through the Spark datasource or the DeltaStreamer.

**Expected behavior**

The write initializes, `tryUpgrade` backfills `hoodie.table.recordkey.fields` via `OneToTwoUpgradeHandler`, and ingestion proceeds. From table version 2 onwards an absent record key should still be reported as a conflict.

**Environment Description**

* Hudi version: 0.x line (`branch-0.x`)
* Storage: S3
* Running on Docker? no

**Additional context**

Only the record-key check has this failure shape. The other validations in `validateTableConfig` (the generic params loop, PreCombineKey, KeyGenerator, PartitionPath, and `validateKeyGeneratorConfig`) all require both sides to be non-null before flagging a conflict, so a table config that a later upgrade handler backfills cannot trip them while it is absent. Partition fields in particular do not fail, despite also being absent at table version 1 and also backfilled by `OneToTwoUpgradeHandler`.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.