orderFields pairs Hive read-column names and ids positionally after de-duplicating them independently
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
## Describe the problem
`HoodieRealtimeRecordReaderUtils.orderFields` de-duplicates Hive's read-column **names** and **ids** independently, then pairs them **positionally**:
```java
Set fieldOrdersSet = new LinkedHashSet<>(Arrays.asList(fieldOrdersWithDups));
String[] fieldOrders = fieldOrdersSet.toArray(new String[0]);
...
Set fieldNamesSet = new LinkedHashSet<>(fieldNames);
String[] fieldNamesArray = fieldNamesSet.toArray(new String[0]);
for (int ox = 0; ox < fieldOrders.length; ox++) {
orderedFieldMap.put(Integer.parseInt(fieldOrders[ox]), fieldNamesArray[ox]);
}
```
That is only sound if duplicates fall at the same offsets on both sides, and Hive guarantees they do not: `ColumnProjectionUtils.appendReadColumns` **prepends** ids (`newConfStr = id + "," + old`) while `appendReadColumnNames` **appends** names. So two accumulation rounds on one `JobConf` can produce lists whose duplicate positions differ, and the counts still match — no exception, silently wrong mapping:
```
READ_COLUMN_NAMES = _hoodie_commit_time,rider,driver,fare
READ_COLUMN_IDS = 2,3,0,1
orderFields -> [driver, fare, _hoodie_commit_time, rider]
```
Hive's own `getReadColumnIDs` documents the hazard: *"some code uses this list to correlate with column names, and yet these lists may contain duplicates, which this call will remove and the other won't."*
## Why this may look harmless today
`projectionFields` feeds `generateProjectionSchema`, and the downstream consumers resolve by name, so the wrong order appears masked. That makes it a latent trap rather than a live data bug as far as I can tell — but it is unverified either way, and it sits in the method that produces the `Error ordering fields for storage read` diagnostics.
## Suggested direction
Pair names and ids before de-duplicating, so a duplicate removes the pair rather than shifting one side relative to the other. Any fix needs a test that produces the prepend/append asymmetry above.
## Context
Raised out of review on #19463, which improves the diagnostics in this method but deliberately does not change the pairing. Filed so the message improvement is not later mistaken for a fix for this.
Credit to @voonhous for identifying the asymmetry and the Hive citation.
## Related
- #19463
- #14673 (HUDI-1286)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with HoodieRealtimeRecordReaderUtils.orderFields and inspect how it consumes Hive's read-column names and ids. Reproduce the prepend/append asymmetry described in the issue, then add a test covering the shown values and verify that de-duplicating paired entries preserves the correct name-to-id mapping.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 70/100