ErrorTableAwareChainedTransformer throws HoodieValidationException when custom transformer drops _corrupt_record
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Bug Description
**What happened:**
Custom JAR transformers that do column projection (e.g. `ColumnFilter` with `mode=include`) drop `_corrupt_record` since they are unaware of the error-table contract. `ErrorTableAwareChainedTransformer` calls `validate()` after every transformer in the chain, throwing `HoodieValidationException: Invalid condition, columnName=_corrupt_record is not present in transformer output schema`.
**What you expected:**
Pipeline should complete successfully — `_corrupt_record` should be re-injected if a transformer drops it.
**Steps to reproduce:**
1. Enable error table (`hoodie.errortable.enable=true`)
2. Configure a custom transformer that does `dataset.select(explicitColumns)` (not including `_corrupt_record`)
3. Run HoodieStreamer
### Environment
**Hudi version:** master (0.16.0-SNAPSHOT)
**Query engine:** Spark 3.5
**Relevant configs:** `hoodie.errortable.enable=true`, `hoodie.errortable.write.class=`
### Logs and Stack Trace
```
org.apache.hudi.exception.HoodieValidationException: Invalid condition, columnName=_corrupt_record is not present in transformer output schema
at org.apache.hudi.utilities.streamer.ErrorTableUtils.validate(ErrorTableUtils.java:88)
at org.apache.hudi.utilities.transform.ErrorTableAwareChainedTransformer.apply(ErrorTableAwareChainedTransformer.java:59)
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with ErrorTableAwareChainedTransformer.java at the apply() call shown in the stack trace, then read ErrorTableUtils.java around validate(). Reproduce with HoodieStreamer, error tables enabled, and a custom transformer that projects columns without _corrupt_record; done means the pipeline completes and _corrupt_record is re-injected rather than causing HoodieValidationException.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, stream-processing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 70/100