apache / apache/hudi

[SUPPORT] How to Suppress the HoodieWriterCommitMessage on each Parquet file it Writes HoodieWriterCommitMessage

Open
#12,854 5 comments 0 reactions 0 assignees View on GitHub
type:refactor
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Describe the problem you faced**
Seeing lot of Messages in the Spark Log File for Every Parquet file the Spark job is Writing to S3 with Hudi Configuration
We are currently on Hudi 0.11.0 and we don't see these log Messages until we enable hoodie.write.commit.callback.on = true but looks like in Hudi 15 , this is default and Call Back Feature is moved to a Different Objects.

INFO internal.DataSourceInternalWriterHelper: Received commit of a data writer = HoodieWriterCommitMessage{writeStatuses=[WriteStatus {fileId=, writeStat=HoodieWriteStat{fileId='', path='.parquet', prevCommit='null', numWrites= , numDeletes=0, numUpdateWrites=0, totalWriteBytes= , totalWriteErrors=0, tempPath='null', cdcStats='null', partitionPath='', totalLogRecords=0, totalLogFilesCompacted=0, totalLogSizeCompacted=0, totalUpdatedRecordsCompacted=0, totalLogBlocks=0, totalCorruptLogBlock=0, totalRollbackBlocks=0}, globalError='null', hasErrors='false', errorCount='0', errorPct='0.0'}]}

**To Reproduce**

Steps to reproduce the behavior:

1. Just Run Any Spark job With hudi 0.15.0 and you will notice this Error

**Environment Description**

* Hudi version : 0.15.0

* Spark version : 3.5.1

* EMR Version : 7.3

* Storage (HDFS/S3/GCS..) : S3

* Running on Docker? (yes/no) :no

**Additional context**
I have tried Disabling following Parameters but this did not help .
hoodie.cleaner.incremental.mode=false
hoodie.embed.timeline.server=false

My Hudi Configs :
class org.apache.hudi.spark3.internal.HoodieDataSourceInternalTable,
hoodie.cleaner.incremental.mode=true,
hoodie.payload.ordering.field=trepprealid,
hoodie.sensitive.config.keys=ssl,tls,sasl,auth,credentials,
hoodie.datasource.write.insert.drop.duplicates=false, hoodie.index.hbase.qps.allocator.class=org.apache.hudi.index.hbase.DefaultHBaseQPSResourceAllocator, hoodie.clustering.plan.strategy.single.group.clustering.enabled=true,
hoodie.memory.merge.fraction=0.6,
hoodie.client.init.callback.classes=,
hoodie.bucket.index.num.buckets=256,
hoodie.datasource.hive_sync.database=presentation_dev,
hoodie.filesystem.view.remote.port=26754,
hoodie.metrics.lock.enable=false,
hoodie.metadata.record.index.max.filegroup.count=10000,
hoodie.global.simple.index.parallelism=100,
hoodie.clustering.schedule.inline=false,
hoodie.combine.before.insert=false,
hoodie.clustering.inline.max.commits=4,
hoodie.fail.writes.on.inline.table.service.exception=true,
hoodie.write.lock.zookeeper.connection_timeout_ms=15000,
hoodie.bloom.index.keys.per.bucket=10000000,
hoodie.write.concurrency.async.conflict.detector.period_ms=30000,
hoodie.datasource.write.row.writer.enable=true,
hoodie.embed.timeline.server=true,
hoodie.parquet.small.file.limit=104857600

**Stacktrace**

```Add the stacktrace of the error.```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the Spark 3.5.1 entry point in org.apache.hudi.spark3.internal.HoodieDataSourceInternalTable and the internal.DataSourceInternalWriterHelper log message for HoodieWriterCommitMessage. Compare the callback behavior between Hudi 0.11.0 and 0.15.0, including the listed callback-related configuration, and verify that the per-file messages can be suppressed without disabling the required callback behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.