linkedin / linkedin/dr-elephant

ERROR [dr-el-executor-thread-1] com.linkedin.drelephant.ElephantRunner : java.lang.IndexOutOfBoundsException: Index: 0, Size: 0

Open
#645 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
1.4k
Forks
839
PR merge metrics
No merged PRs in 30d

Description

Hi Team,

We can able to see Mapreduce, Hive & Spark jobs in logs as well as Dashboard UI. but along with application we see some error.

find the error in dr_elephant.log.
12-31-2019 16:37:29 ERROR [dr-el-executor-thread-1] com.linkedin.drelephant.ElephantRunner : http://server2.com:8188/ws/v1/timeline/TEZ_APPLICATION/tez_application_1577712018820_0004
12-31-2019 16:37:29 ERROR [dr-el-executor-thread-1] com.linkedin.drelephant.ElephantRunner : java.io.FileNotFoundException: http://server2.com:8188/ws/v1/timeline/TEZ_APPLICATION/tez_application_1577712018820_0004
12-31-2019 16:37:29 ERROR [dr-el-executor-thread-0] com.linkedin.drelephant.ElephantRunner : Index: 0, Size: 0
12-31-2019 16:37:29 ERROR [dr-el-executor-thread-0] com.linkedin.drelephant.ElephantRunner : java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
12-31-2019 16:37:29 ERROR [Auto Tuner Thread] com.linkedin.drelephant.tuning.PSOParamGenerator : No auto-tuning enabled jobs found
12-31-2019 16:38:29 ERROR [Auto Tuner Thread] com.linkedin.drelephant.tuning.PSOParamGenerator : No auto-tuning enabled jobs found
12-31-2019 16:39:29 ERROR [dr-el-executor-thread-1] com.linkedin.drelephant.ElephantRunner : Index: 0, Size: 0
12-31-2019 16:39:29 ERROR [dr-el-executor-thread-1] com.linkedin.drelephant.ElephantRunner : java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
12-31-2019 16:39:29 ERROR [Auto Tuner Thread] com.linkedin.drelephant.tuning.PSOParamGenerator : No auto-tuning enabled jobs found
[hdfs@sjdcdlake06 elephant]$

Find the error in log dr.log
[error] o.a.s.s.ReplayListenerBus - Exception parsing Spark event log: application_1577712018820_0291
scala.MatchError: org.apache.spark.sql.execution.ui.SparkListenerSQLExecutionStart (of class java.lang.String)
at org.apache.spark.util.JsonProtocol$.sparkEventFromJson(JsonProtocol.scala:466) ~[org.apache.spark.spark-core_2.10-1.4.0.jar:1.4.0]
at org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:58) ~[org.apache.spark.spark-core_2.10-1.4.0.jar:1.4.0]
at org.apache.spark.deploy.history.SparkDataCollection.load(SparkDataCollection.scala:298) [com.linkedin.drelephant.dr-elephant-2.1.7.jar:2.1.7]
at com.linkedin.drelephant.spark.fetchers.SparkRestClient$$anonfun$fetchEventLogAndParse$2.apply(SparkRestClient.scala:147) [com.linkedin.drelephant.dr-elephant-2.1.7.jar:2.1.7]
at com.linkedin.drelephant.spark.fetchers.SparkRestClient$$anonfun$fetchEventLogAndParse$2.apply(SparkRestClient.scala:142) [com.linkedin.drelephant.dr-elephant-2.1.7.jar:2.1.7]
at resource.AbstractManagedResource$$anonfun$5.apply(AbstractManagedResource.scala:86) [com.jsuereth.scala-arm_2.10-1.4.jar:1.4]
[error] o.a.s.s.ReplayListenerBus - Malformed line #5: {"Event":"org.apache.spark.sql.execution.ui.SparkListenerSQLExecutionStart","executionId":0,"description":"createOrReplaceTempView at srcDataDef.scala:48","details":"org.apache.spark.sql.Dataset.createOrReplaceTempView(Dataset.scala:3038)\nsrcDataDef$.main(srcDataDef.scala:48)\nsrcDataDef.main(srcDataDef.scala)\nsun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\nsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\nsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\njava.lang.reflect.Method.invoke(Method.java:498)\norg.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)\norg.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:904)\norg.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)\norg.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)\norg.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)\norg.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)","physicalPlanDescription":"== Parsed Logical Plan ==\nCreateViewCommand conn_meta, false, true, LocalTempView\n +- AnalysisBarrier\n

Also unable to see Application job type for mapreduce in dashboard UI. attached snapshot herewith
dr_elephant.log

is there any setting to see HIVE LLAP jobs details in Dr. Elephant?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the logged failures in ElephantRunner, PSOParamGenerator, SparkRestClient.scala, and SparkDataCollection.scala, using the attached dr_elephant.log and dr.log as the reproduction evidence. Determine which errors prevent MapReduce application types and Spark events from appearing in the dashboard; done means the reported exceptions are handled and the affected job details display correctly.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, java, spark
Domain
backend, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.