[SUPPORT] High runtime for a batch in SparkWriteHelper stage
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**Describe the problem you faced**
We are observing higher run times for a batch , it took 15hr plus to complete single batch, the subsequent batches are running fine. The dataset in question is not big. Attaching few screenshots for reference, GC times are less.
hoodieConfigs for reference

**To Reproduce**
Steps to reproduce the behavior:
1.
2.
3.
4.
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment Description**
* Hudi version : 0.10.1
* Spark version : 3.0.3
* Hive version : 3.1.2
* Hadoop version : 3.2.2
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) : NO
**Additional context**
Hudi Configs
```
hoodieConfigs:
hoodie.datasource.write.operation: upsert
hoodie.datasource.write.table.type: MERGE_ON_READ
hoodie.datasource.write.partitionpath.field: ""
hoodie.datasource.write.keygenerator.class: org.apache.hudi.keygen.NonpartitionedKeyGenerator
hoodie.metrics.on: true
hoodie.metrics.reporter.type: CLOUDWATCH
hoodie.datasource.hive_sync.partition_extractor_class: org.apache.hudi.hive.NonPartitionedExtractor
hoodie.parquet.max.file.size: 6110612736
hoodie.compact.inline: true
hoodie.clean.automatic: true
hoodie.compact.inline.trigger.strategy: NUM_AND_TIME
hoodie.clean.async: true
hoodie.cleaner.policy: KEEP_LATEST_COMMITS
hoodie.cleaner.commits.retained: 120
hoodie.keep.min.commits: 130
hoodie.keep.max.commits: 131
```
Spark Job configs
```
{
"className": "com.hotstar.driver.CdcCombinedDriver",
"proxyUser": "root",
"driverCores": 1,
"executorCores": 4,
"executorMemory": "4G",
"driverMemory": "4G",
"queue": "cdc",
"name": "hudiJob",
"file": "s3a://bucket/jars/prod.jar",
"conf": {
"spark.eventLog.enabled": "false",
"spark.ui.enabled": "true",
"spark.streaming.concurrentJobs": "1",
"spark.streaming.backpressure.enabled": "false",
"spark.streaming.kafka.maxRatePerPartition": "500",
"spark.yarn.am.nodeLabelExpression": "cdc",
"spark.shuffle.service.enabled": "true",
"spark.driver.maxResultSize": "8g",
"spark.driver.memoryOverhead": "2048",
"spark.executor.memoryOverhead": "2048",
"spark.dynamicAllocation.enabled": "true",
"spark.dynamicAllocation.minExecutors": "25",
"spark.dynamicAllocation.maxExecutors": "50",
"spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version": "2",
"spark.jars.packages": "org.apache.spark:spark-avro_2.12:3.0.2,com.izettle:metrics-influxdb:1.2.3",
"spark.serializer": "org.apache.spark.serializer.KryoSerializer",
"spark.rdd.compress": "true",
"spark.sql.hive.convertMetastoreParquet": "false",
"spark.yarn.maxAppAttempts": "1",
"spark.task.cpus": "1"
}
}
```
**Stacktrace**
```Add the stacktrace of the error.```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.