[SUPPORT] How to Suppress the HoodieWriterCommitMessage on each Parquet file it Writes HoodieWriterCommitMessage
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**Describe the problem you faced**
Seeing lot of Messages in the Spark Log File for Every Parquet file the Spark job is Writing to S3 with Hudi Configuration
We are currently on Hudi 0.11.0 and we don't see these log Messages until we enable hoodie.write.commit.callback.on = true but looks like in Hudi 15 , this is default and Call Back Feature is moved to a Different Objects.
INFO internal.DataSourceInternalWriterHelper: Received commit of a data writer = HoodieWriterCommitMessage{writeStatuses=[WriteStatus {fileId=, writeStat=HoodieWriteStat{fileId='', path='.parquet', prevCommit='null', numWrites= , numDeletes=0, numUpdateWrites=0, totalWriteBytes= , totalWriteErrors=0, tempPath='null', cdcStats='null', partitionPath='', totalLogRecords=0, totalLogFilesCompacted=0, totalLogSizeCompacted=0, totalUpdatedRecordsCompacted=0, totalLogBlocks=0, totalCorruptLogBlock=0, totalRollbackBlocks=0}, globalError='null', hasErrors='false', errorCount='0', errorPct='0.0'}]}
**To Reproduce**
Steps to reproduce the behavior:
1. Just Run Any Spark job With hudi 0.15.0 and you will notice this Error
**Environment Description**
* Hudi version : 0.15.0
* Spark version : 3.5.1
* EMR Version : 7.3
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) :no
**Additional context**
I have tried Disabling following Parameters but this did not help .
hoodie.cleaner.incremental.mode=false
hoodie.embed.timeline.server=false
My Hudi Configs :
class org.apache.hudi.spark3.internal.HoodieDataSourceInternalTable,
hoodie.cleaner.incremental.mode=true,
hoodie.payload.ordering.field=trepprealid,
hoodie.sensitive.config.keys=ssl,tls,sasl,auth,credentials,
hoodie.datasource.write.insert.drop.duplicates=false, hoodie.index.hbase.qps.allocator.class=org.apache.hudi.index.hbase.DefaultHBaseQPSResourceAllocator, hoodie.clustering.plan.strategy.single.group.clustering.enabled=true,
hoodie.memory.merge.fraction=0.6,
hoodie.client.init.callback.classes=,
hoodie.bucket.index.num.buckets=256,
hoodie.datasource.hive_sync.database=presentation_dev,
hoodie.filesystem.view.remote.port=26754,
hoodie.metrics.lock.enable=false,
hoodie.metadata.record.index.max.filegroup.count=10000,
hoodie.global.simple.index.parallelism=100,
hoodie.clustering.schedule.inline=false,
hoodie.combine.before.insert=false,
hoodie.clustering.inline.max.commits=4,
hoodie.fail.writes.on.inline.table.service.exception=true,
hoodie.write.lock.zookeeper.connection_timeout_ms=15000,
hoodie.bloom.index.keys.per.bucket=10000000,
hoodie.write.concurrency.async.conflict.detector.period_ms=30000,
hoodie.datasource.write.row.writer.enable=true,
hoodie.embed.timeline.server=true,
hoodie.parquet.small.file.limit=104857600
**Stacktrace**
```Add the stacktrace of the error.```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the Spark 3.5.1 entry point in org.apache.hudi.spark3.internal.HoodieDataSourceInternalTable and the internal.DataSourceInternalWriterHelper log message for HoodieWriterCommitMessage. Compare the callback behavior between Hudi 0.11.0 and 0.15.0, including the listed callback-related configuration, and verify that the per-file messages can be suppressed without disabling the required callback behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100