NullPointerException when analyzing Spark Job

Open

#426 opened on Aug 27, 2018 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language: Java
Stars: 1.4k
Forks: 839
PR merge metrics: No merged PRs in 30d

Description

when processing jobs, dr-elephant analysis tasks crash with below
08-27-2018 10:16:05 ERROR [dr-el-executor-thread-2] com.linkedin.drelephant.ElephantRunner : java.lang.NullPointerException
at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1838)
at sun.misc.FloatingDecimal.parseDouble(FloatingDecimal.java:110)
at java.lang.Double.parseDouble(Double.java:538)
at com.linkedin.drelephant.tez.TezMetricsAggregator.aggregate(TezMetricsAggregator.java:58)
at com.linkedin.drelephant.analysis.AnalyticJob.getAnalysis(AnalyticJob.java:275)
at com.linkedin.drelephant.ElephantRunner$ExecutorJob.run(ElephantRunner.java:175)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

08-27-2018 10:16:05 WARN [dr-el-executor-thread-2] com.linkedin.drelephant.ElephantRunner : Add analytic job id [application_1535259914264_2461] into the retry list.
08-27-2018 10:16:05 INFO [dr-el-executor-thread-2] com.linkedin.drelephant.analysis.AnalyticJobGeneratorHadoop2 : Retry queue size is 2
08-27-2018 10:16:10 ERROR [dr-el-executor-thread-1] com.linkedin.drelephant.ElephantRunner :
08-27-2018 10:16:10 ERROR [dr-el-executor-thread-1] com.linkedin.drelephant.ElephantRunner : java.lang.NullPointerException
at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1838)
at sun.misc.FloatingDecimal.parseDouble(FloatingDecimal.java:110)
at java.lang.Double.parseDouble(Double.java:538)
at com.linkedin.drelephant.tez.TezMetricsAggregator.aggregate(TezMetricsAggregator.java:58)
at com.linkedin.drelephant.analysis.AnalyticJob.getAnalysis(AnalyticJob.java:275)
at com.linkedin.drelephant.ElephantRunner$ExecutorJob.run(ElephantRunner.java:175)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

08-27-2018 10:16:10 WARN [dr-el-executor-thread-1] com.linkedin.drelephant.ElephantRunner : Add analytic job id [application_1535259914264_1277] into the retry list.
08-27-2018 10:16:10 INFO [dr-el-executor-thread-1] com.linkedin.drelephant.analysis.AnalyticJobGeneratorHadoop2 : Retry queue size is 3

Contributor guide

No contributing guide indexed for this repository

First steps

Read the whole issue, then the project's contributing guide.
Comment on the issue to say you are picking it up — it saves two people doing the same work.
Fork the repository and make your change on a branch.
Open a pull request that references the issue number.

Research direction

Start at TezMetricsAggregator.java line 58, where the stack trace shows the failure, and trace the analysis flow from AnalyticJob.java line 275. Reproduce the Spark job analysis using the reported failure pattern and verify that the affected job completes without being added to the retry list.

Written by the indexing model from the issue text.

Assessment

Tech stack: java, spark
Domain: data-engineering, distributed-systems
Issue type: Bug
Difficulty: 3/5
Estimated time: 1-2 days
Activity status: Stale
Clarity: Mostly clear
Newbie friendliness: 35/100