[SUPPORT] spark task execute too long and can not finish when ObjectSizeCalculator.getObjectSize
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**_Tips before filing an issue_**
**Describe the problem you faced**
like #10504 , in different func
```shell
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:502)
java.lang.UNIXProcess.waitFor(UNIXProcess.java:396)
org.apache.hudi.org.openjdk.jol.vm.sa.ServiceabilityAgentSupport.callAgent(ServiceabilityAgentSupport.java:190)
org.apache.hudi.org.openjdk.jol.vm.sa.ServiceabilityAgentSupport.callAgent(ServiceabilityAgentSupport.java:163)
org.apache.hudi.org.openjdk.jol.vm.sa.ServiceabilityAgentSupport.getUniverseData(ServiceabilityAgentSupport.java:301)
org.apache.hudi.org.openjdk.jol.vm.VM.current(VM.java:77)
org.apache.hudi.org.openjdk.jol.info.GraphWalker.walk(GraphWalker.java:97)
org.apache.hudi.org.openjdk.jol.info.GraphLayout.parseInstance(GraphLayout.java:54)
org.apache.hudi.common.util.ObjectSizeCalculator.getObjectSize(ObjectSizeCalculator.java:57)
org.apache.hudi.common.util.HoodieRecordSizeEstimator.(HoodieRecordSizeEstimator.java:40)
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner.(HoodieMergedLogRecordScanner.java:107)
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner.(HoodieMergedLogRecordScanner.java:74)
org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner$Builder.build(HoodieMergedLogRecordScanner.java:465)
org.apache.hudi.LogFileIterator$.$anonfun$scanLog$1(Iterators.scala:329)
org.apache.hudi.LogFileIterator$$$Lambda$1054/69444513.apply(Unknown Source)
org.apache.spark.sql.hive.HadoopUgiUtils$$anon$1.run(HadoopUgiUtils.scala:54)
java.security.AccessController.doPrivileged(Native Method)
javax.security.auth.Subject.doAs(Subject.java:422)
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1989)
org.apache.spark.sql.hive.HadoopUgiUtils$.doAsWithHiveSuperUser(HadoopUgiUtils.scala:53)
org.apache.hudi.LogFileIterator$.scanLog(Iterators.scala:261)
org.apache.hudi.LogFileIterator.(Iterators.scala:93)
org.apache.hudi.RecordMergingFileIterator.(Iterators.scala:173)
org.apache.hudi.HoodieMergeOnReadRDD.compute(HoodieMergeOnReadRDD.scala:100)
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:369)
org.apache.spark.rdd.RDD.iterator(RDD.scala:333)
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:369)
org.apache.spark.rdd.RDD.iterator(RDD.scala:333)
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:369)
org.apache.spark.rdd.RDD.iterator(RDD.scala:333)
org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
org.apache.spark.scheduler.Task.run(Task.scala:131)
org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506)
org.apache.spark.executor.Executor$TaskRunner$$Lambda$476/220247377.apply(Unknown Source)
org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1463)
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
```
**To Reproduce**
can not reporduce
**Expected behavior**
**Environment Description**
* Hudi version : 0.13.1
* Spark version : 3.2
* Hive version :
* Hadoop version :
* Storage (HDFS/S3/GCS..) :
* Running on Docker? (yes/no) :
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with org.apache.hudi.common.util.ObjectSizeCalculator.getObjectSize and the stack path through HoodieRecordSizeEstimator and HoodieMergedLogRecordScanner. Review how the Spark task reaches this code and investigate the reported wait in ServiceabilityAgentSupport. The issue has no reproducible case, so done should include a confirmed cause and evidence that the task can finish normally.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100