apache / apache/hudi

[SUPPORT] spark task execute too long and can not finish when ObjectSizeCalculator.getObjectSize

Open
#11,879 13 comments 0 reactions 0 assignees View on GitHub
area:writer priority:critical
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**_Tips before filing an issue_**

**Describe the problem you faced**

like #10504 , in different func

```shell
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:502)
java.lang.UNIXProcess.waitFor(UNIXProcess.java:396)
org.apache.hudi.org.openjdk.jol.vm.sa.ServiceabilityAgentSupport.callAgent(ServiceabilityAgentSupport.java:190)
org.apache.hudi.org.openjdk.jol.vm.sa.ServiceabilityAgentSupport.callAgent(ServiceabilityAgentSupport.java:163)
org.apache.hudi.org.openjdk.jol.vm.sa.ServiceabilityAgentSupport.getUniverseData(ServiceabilityAgentSupport.java:301)
org.apache.hudi.org.openjdk.jol.vm.VM.current(VM.java:77)
org.apache.hudi.org.openjdk.jol.info.GraphWalker.walk(GraphWalker.java:97)
org.apache.hudi.org.openjdk.jol.info.GraphLayout.parseInstance(GraphLayout.java:54)
org.apache.hudi.common.util.ObjectSizeCalculator.getObjectSize(ObjectSizeCalculator.java:57)
org.apache.hudi.common.util.HoodieRecordSizeEstimator.(HoodieRecordSizeEstimator.java:40)
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner.(HoodieMergedLogRecordScanner.java:107)
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner.(HoodieMergedLogRecordScanner.java:74)
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner$Builder.build(HoodieMergedLogRecordScanner.java:465)
org.apache.hudi.LogFileIterator$.$anonfun$scanLog$1(Iterators.scala:329)
org.apache.hudi.LogFileIterator$$$Lambda$1054/69444513.apply(Unknown Source)
org.apache.spark.sql.hive.HadoopUgiUtils$$anon$1.run(HadoopUgiUtils.scala:54)
java.security.AccessController.doPrivileged(Native Method)
javax.security.auth.Subject.doAs(Subject.java:422)
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1989)
org.apache.spark.sql.hive.HadoopUgiUtils$.doAsWithHiveSuperUser(HadoopUgiUtils.scala:53)
org.apache.hudi.LogFileIterator$.scanLog(Iterators.scala:261)
org.apache.hudi.LogFileIterator.(Iterators.scala:93)
org.apache.hudi.RecordMergingFileIterator.(Iterators.scala:173)
org.apache.hudi.HoodieMergeOnReadRDD.compute(HoodieMergeOnReadRDD.scala:100)
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:369)
org.apache.spark.rdd.RDD.iterator(RDD.scala:333)
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:369)
org.apache.spark.rdd.RDD.iterator(RDD.scala:333)
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:369)
org.apache.spark.rdd.RDD.iterator(RDD.scala:333)
org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
org.apache.spark.scheduler.Task.run(Task.scala:131)
org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506)
org.apache.spark.executor.Executor$TaskRunner$$Lambda$476/220247377.apply(Unknown Source)
org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1463)
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
```

**To Reproduce**

can not reporduce

**Expected behavior**

**Environment Description**

* Hudi version : 0.13.1

* Spark version : 3.2

* Hive version :

* Hadoop version :

* Storage (HDFS/S3/GCS..) :

* Running on Docker? (yes/no) :

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with org.apache.hudi.common.util.ObjectSizeCalculator.getObjectSize and the stack path through HoodieRecordSizeEstimator and HoodieMergedLogRecordScanner. Review how the Spark task reaches this code and investigate the reported wait in ServiceabilityAgentSupport. The issue has no reproducible case, so done should include a confirmed cause and evidence that the task can finish normally.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.