linkedin / linkedin/dr-elephant
ERROR [dr-el-executor-thread-1] com.linkedin.drelephant.ElephantRunner : java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 1.4k
- Forks
- 839
- PR merge metrics
- No merged PRs in 30d
Description
Hi Team,
We can able to see Mapreduce, Hive & Spark jobs in logs as well as Dashboard UI. but along with application we see some error.
find the error in dr_elephant.log.
12-31-2019 16:37:29 ERROR [dr-el-executor-thread-1] com.linkedin.drelephant.ElephantRunner : http://server2.com:8188/ws/v1/timeline/TEZ_APPLICATION/tez_application_1577712018820_0004
12-31-2019 16:37:29 ERROR [dr-el-executor-thread-1] com.linkedin.drelephant.ElephantRunner : java.io.FileNotFoundException: http://server2.com:8188/ws/v1/timeline/TEZ_APPLICATION/tez_application_1577712018820_0004
12-31-2019 16:37:29 ERROR [dr-el-executor-thread-0] com.linkedin.drelephant.ElephantRunner : Index: 0, Size: 0
12-31-2019 16:37:29 ERROR [dr-el-executor-thread-0] com.linkedin.drelephant.ElephantRunner : java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
12-31-2019 16:37:29 ERROR [Auto Tuner Thread] com.linkedin.drelephant.tuning.PSOParamGenerator : No auto-tuning enabled jobs found
12-31-2019 16:38:29 ERROR [Auto Tuner Thread] com.linkedin.drelephant.tuning.PSOParamGenerator : No auto-tuning enabled jobs found
12-31-2019 16:39:29 ERROR [dr-el-executor-thread-1] com.linkedin.drelephant.ElephantRunner : Index: 0, Size: 0
12-31-2019 16:39:29 ERROR [dr-el-executor-thread-1] com.linkedin.drelephant.ElephantRunner : java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
12-31-2019 16:39:29 ERROR [Auto Tuner Thread] com.linkedin.drelephant.tuning.PSOParamGenerator : No auto-tuning enabled jobs found
[hdfs@sjdcdlake06 elephant]$
Find the error in log dr.log
[error] o.a.s.s.ReplayListenerBus - Exception parsing Spark event log: application_1577712018820_0291
scala.MatchError: org.apache.spark.sql.execution.ui.SparkListenerSQLExecutionStart (of class java.lang.String)
at org.apache.spark.util.JsonProtocol$.sparkEventFromJson(JsonProtocol.scala:466) ~[org.apache.spark.spark-core_2.10-1.4.0.jar:1.4.0]
at org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:58) ~[org.apache.spark.spark-core_2.10-1.4.0.jar:1.4.0]
at org.apache.spark.deploy.history.SparkDataCollection.load(SparkDataCollection.scala:298) [com.linkedin.drelephant.dr-elephant-2.1.7.jar:2.1.7]
at com.linkedin.drelephant.spark.fetchers.SparkRestClient$$anonfun$fetchEventLogAndParse$2.apply(SparkRestClient.scala:147) [com.linkedin.drelephant.dr-elephant-2.1.7.jar:2.1.7]
at com.linkedin.drelephant.spark.fetchers.SparkRestClient$$anonfun$fetchEventLogAndParse$2.apply(SparkRestClient.scala:142) [com.linkedin.drelephant.dr-elephant-2.1.7.jar:2.1.7]
at resource.AbstractManagedResource$$anonfun$5.apply(AbstractManagedResource.scala:86) [com.jsuereth.scala-arm_2.10-1.4.jar:1.4]
[error] o.a.s.s.ReplayListenerBus - Malformed line #5: {"Event":"org.apache.spark.sql.execution.ui.SparkListenerSQLExecutionStart","executionId":0,"description":"createOrReplaceTempView at srcDataDef.scala:48","details":"org.apache.spark.sql.Dataset.createOrReplaceTempView(Dataset.scala:3038)\nsrcDataDef$.main(srcDataDef.scala:48)\nsrcDataDef.main(srcDataDef.scala)\nsun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\nsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\nsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\njava.lang.reflect.Method.invoke(Method.java:498)\norg.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)\norg.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:904)\norg.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)\norg.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)\norg.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)\norg.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)","physicalPlanDescription":"== Parsed Logical Plan ==\nCreateViewCommand conn_meta, false, true, LocalTempView\n +- AnalysisBarrier\n
Also unable to see Application job type for mapreduce in dashboard UI. attached snapshot herewith
dr_elephant.log
is there any setting to see HIVE LLAP jobs details in Dr. Elephant?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the logged failures in ElephantRunner, PSOParamGenerator, SparkRestClient.scala, and SparkDataCollection.scala, using the attached dr_elephant.log and dr.log as the reproduction evidence. Determine which errors prevent MapReduce application types and Spark events from appearing in the dashboard; done means the reported exceptions are handled and the affected job details display correctly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, java, spark
- Domain
- backend, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100