apache / apache/hudi

[SUPPORT] What Class Name to use for hoodie.errortable.write.class

Open
#11,252 9 comments 0 reactions 0 assignees View on GitHub
type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

I'm trying out Hudi error tables, but I'm having trouble finding the documentation for the hoodie.errortable.write.class value. Could you please assist me?

# sample config
```
hoodie.datasource.write.keygenerator.class=org.apache.hudi.keygen.SimpleKeyGenerator
hoodie.datasource.write.recordkey.field=invoiceid
hoodie.datasource.write.partitionpath.field=destinationstate
hoodie.streamer.source.dfs.root=file:///Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E1/sampledata/
hoodie.datasource.write.precombine.field=replicadmstimestamp
hoodie.streamer.transformer.sql=SELECT * FROM a where sas
hoodie.errortable.base.path=file:///Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E1/error/
hoodie.errortable.target.table.name=error_invoice
hoodie.errortable.enable=true
hoodie.errortable.write.class=

```

# Job
```

spark-submit \
--class org.apache.hudi.utilities.streamer.HoodieStreamer \
--packages org.apache.hudi:hudi-spark3.4-bundle_2.12:0.14.0 \
--properties-file spark-config.properties \
--master 'local[*]' \
--executor-memory 1g \
/Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E1/jar/hudi-utilities-slim-bundle_2.12-0.14.0.jar \
--table-type COPY_ON_WRITE \
--op UPSERT \
--transformer-class org.apache.hudi.utilities.transform.SqlQueryBasedTransformer \
--source-ordering-field replicadmstimestamp \
--source-class org.apache.hudi.utilities.sources.ParquetDFSSource \
--target-base-path file:///Users/soumilshah/IdeaProjects/SparkProject/apache-hudi-delta-streamer-labs/E1/silver/ \
--target-table invoice \
--props hudi_tbl.props
```

I want to purposely fail the job and I want to see error tables being created

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the `hoodie.errortable.write.class` configuration in the sample properties and the `HoodieStreamer` entry point shown in the job command. Determine which class name is supported and document how intentional job failures produce error tables; the issue does not identify a documentation file or test to update.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, documentation
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.