apache / apache/hudi

orderFields pairs Hive read-column names and ids positionally after de-duplicating them independently

Open
#19,506 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

## Describe the problem

`HoodieRealtimeRecordReaderUtils.orderFields` de-duplicates Hive's read-column **names** and **ids** independently, then pairs them **positionally**:

```java
Set fieldOrdersSet = new LinkedHashSet<>(Arrays.asList(fieldOrdersWithDups));
String[] fieldOrders = fieldOrdersSet.toArray(new String[0]);
...
Set fieldNamesSet = new LinkedHashSet<>(fieldNames);
String[] fieldNamesArray = fieldNamesSet.toArray(new String[0]);
for (int ox = 0; ox < fieldOrders.length; ox++) {
orderedFieldMap.put(Integer.parseInt(fieldOrders[ox]), fieldNamesArray[ox]);
}
```

That is only sound if duplicates fall at the same offsets on both sides, and Hive guarantees they do not: `ColumnProjectionUtils.appendReadColumns` **prepends** ids (`newConfStr = id + "," + old`) while `appendReadColumnNames` **appends** names. So two accumulation rounds on one `JobConf` can produce lists whose duplicate positions differ, and the counts still match — no exception, silently wrong mapping:

```
READ_COLUMN_NAMES = _hoodie_commit_time,rider,driver,fare
READ_COLUMN_IDS = 2,3,0,1
orderFields -> [driver, fare, _hoodie_commit_time, rider]
```

Hive's own `getReadColumnIDs` documents the hazard: *"some code uses this list to correlate with column names, and yet these lists may contain duplicates, which this call will remove and the other won't."*

## Why this may look harmless today

`projectionFields` feeds `generateProjectionSchema`, and the downstream consumers resolve by name, so the wrong order appears masked. That makes it a latent trap rather than a live data bug as far as I can tell — but it is unverified either way, and it sits in the method that produces the `Error ordering fields for storage read` diagnostics.

## Suggested direction

Pair names and ids before de-duplicating, so a duplicate removes the pair rather than shifting one side relative to the other. Any fix needs a test that produces the prepend/append asymmetry above.

## Context

Raised out of review on #19463, which improves the diagnostics in this method but deliberately does not change the pairing. Filed so the message improvement is not later mistaken for a fix for this.

Credit to @voonhous for identifying the asymmetry and the Hive citation.

## Related

- #19463
- #14673 (HUDI-1286)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with HoodieRealtimeRecordReaderUtils.orderFields and inspect how it consumes Hive's read-column names and ids. Reproduce the prepend/append asymmetry described in the issue, then add a test covering the shown values and verify that de-duplicating paired entries preserves the correct name-to-id mapping.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
70/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.