apache / apache/hudi

[HoodieStreamer] Register SparkWriteErrorValidator as default pre-commit validator

Open
#18,841 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

## Summary

As part of #18765, `HoodieStreamerWriteStatusValidator`'s error-check logic was migrated into explicit pre-commit orchestration steps in `StreamSync`, and a new `SparkWriteErrorValidator` was added to the validator framework as an opt-in.

The next step is to register `SparkWriteErrorValidator` as a default validator for `HoodieStreamer`, making it the canonical path.

## Before doing so, we must ensure:
1. The hot path (no-validator config) does not error out due to default registration.
2. The Step 4 inline gate in `StreamSync` can then be retired once the default is stable.

## Target
Next minor release.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the validator framework and the StreamSync orchestration changes from #18765, focusing on SparkWriteErrorValidator and the Step 4 inline gate. Verify that registering it by default does not break the no-validator hot path. Done means HoodieStreamer uses the default validator safely, with the inline gate ready to be retired once stability is confirmed.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, stream-processing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.