Upgrading a table from version 1 drops the ordering field from the table config
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Upgrading a table that is still at table version 1 leaves it with no ordering field recorded in `hoodie.properties`, even when the writer is configured with one.
`hoodie.table.precombine.field` is only written at table creation, by `HoodieTableMetaClient.PropertyBuilder`, and only started being written in 0.8.0 (commit `0d8a4d0a56dc`, "Honor ordering field for MOR Spark datasource reader"). A table created before that has no ordering field in its table config, and none of the handlers in the version 1 to 6 upgrade chain adds one:
| Handler | Table properties written |
| --- | --- |
| `OneToTwoUpgradeHandler` | `PARTITION_FIELDS`, `RECORDKEY_FIELDS`, `BASE_FILE_FORMAT` |
| `TwoToThreeUpgradeHandler` | `URL_ENCODE_PARTITIONING`, `HIVE_STYLE_PARTITIONING_ENABLE`, `KEY_GENERATOR_CLASS_NAME` |
| `ThreeToFourUpgradeHandler` | `DATABASE_NAME`, `TABLE_CHECKSUM`, `TABLE_METADATA_PARTITIONS` |
| `FourToFiveUpgradeHandler` | none |
| `FiveToSixUpgradeHandler` | none |
`OneToTwoUpgradeHandler` is the handler that backfills the other key-schema properties from the write config, so the ordering field looks like an omission from that list.
Nothing reports it afterwards. `HoodieWriterUtils.validateTableConfig` flags an ordering field only when the datasource value and the table config value are both non-null and differ, unlike the record key check just above it, which fails when the write config has a key and the table config does not.
The write path itself is unaffected while a writer is configured, since `ConfigUtils.getOrderingField` prefers `hoodie.payload.ordering.field` and then `hoodie.datasource.write.precombine.field` before falling back to the table config. What breaks is everything that only has `hoodie.properties` to go on:
- Copy-on-write updates: `HoodieMergeHandle` calls `DefaultHoodieRecordPayload.combineAndGetUpdateValue`, whose `needUpdatingPersistedRecord` gets a null ordering field and stops comparing, so an older record can overwrite a newer one and the wrong value is written into the base file.
- Spark SQL DML: `ProvidesHoodieConfig` resolves the ordering field from `HoodieTableConfig.PRECOMBINE_FIELD` first, then the datasource key, then `""`, and a SQL session has no write config to fall back on.
- Merge-on-read snapshot reads, which lose ordering in the log merge.
- Flink, through `HoodieTableFactory` and `CompactionUtil.setPreCombineField`.
On copy-on-write the failure is silent: there is no wrong query result to notice, only the wrong record winning an update.
Note that this is not simply "table version 1 predates the property". 0.8.0 still stamped new tables as `HoodieTableVersion.ONE`, so version 1 straddles the change: a version 1 table created by 0.6.0 or 0.7.0 has no ordering field, one created by 0.8.0 does. The presence of the key has to be checked rather than inferred from the table version.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with OneToTwoUpgradeHandler and compare its backfilled properties with HoodieTableMetaClient.PropertyBuilder, checking how the ordering field is represented in hoodie.properties during version 1 to 2 upgrades. Trace HoodieWriterUtils.validateTableConfig and the named COW, Spark SQL DML, MOR, and Flink paths, then add regression coverage showing that a missing ordering field is preserved or recovered correctly for affected readers and writers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, databases, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 58/100