Records with a null ordering field value bypass the error table instead of being quarantined
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**Bug Description**
When Hudi Streamer runs with an error table enabled and an ordering (precombine) field configured, a record whose ordering field value is null is not routed to the error table as a `RECORD_CREATION` failure. It is accepted into the write with an unvalidated ordering value.
`HoodieStreamerUtils.createHoodieRecords` wraps record construction in a try/catch that turns a failure into an `ErrorEvent` with `ErrorEvent.ErrorReason.RECORD_CREATION`, so anything that throws there is quarantined and ingestion continues. A null ordering value does not throw there on either of the two record-creation paths:
- **File-group-reader path** (`requiresPayload == false`, the default since `hoodie.write.merge.handle.class` defaults to `FileGroupReaderBasedMergeHandle`): `HoodieRecordUtils.createHoodieRecord` builds a `HoodieAvroIndexedRecord` that stores the ordering value without validating it.
- **Payload path** (`requiresPayload == true`): `BaseAvroPayload` does have a null-ordering guard, but it is unreachable, because `HoodieRecordUtils.loadPayload(String, GenericRecord, Comparable)` opens with
```java
if (orderingValue == null) {
return loadPayload(recordPayloadClass, record);
}
```
which routes to the record-only overload, so the `(GenericRecord, Comparable)` constructor carrying the throw is never invoked.
The null is therefore only forced later, during the index/write stage, which sits outside the error table's catch. A bare `compareTo` on the ordering value there (for example `BufferedRecordMergerFactory.shouldKeepNewerRecord`) fails the write instead of parking the offending record.
**Expected behavior:** a record that cannot supply a required ordering value is a record-creation failure and belongs in the error table, so ingestion survives one bad record. That is the contract the error table's `RECORD_CREATION` reason already implies, and it is what the 0.x line does, where the same class of failure throws inside the record-creation catch and is quarantined.
**Environment**
- Hudi version: master (1.3.0-SNAPSHOT)
- Spark version: 3.5
- Running on Docker? no
**Logs and Stack Trace**
A test added in the companion PR pins the intended behavior. Against master today it fails on both record-creation paths with the record flowing through unquarantined rather than landing in the error table:
```
expected: <[]> but was: <[HoodieRecord{key=HoodieKey(recordKey=key1, partitionPath=path1), currentLocation='null', newLocation='null'}]>
```
With a reject-on-null added at record creation, both paths quarantine and the test passes.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at HoodieStreamerUtils.createHoodieRecords and trace both HoodieRecordUtils.createHoodieRecord paths through BaseAvroPayload; inspect BufferedRecordMergerFactory for where the null is currently forced. Run the companion test described in the issue, and confirm that null ordering values on both paths become RECORD_CREATION error-table entries while ingestion continues.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, stream-processing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 66/100