linkedin / linkedin/dr-elephant

Elephant Runner fails on SPARK analysis with new Fetcher/Metrics/Heuristics

Open
#176 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
1.4k
Forks
839
PR merge metrics
No merged PRs in 30d

Description

```
12-20-2016 13:15:31 ERROR [ForkJoinPool-1-worker-1] com.linkedin.drelephant.spark.fetchers.SparkRestClient : error reading http://demo-nn1.company.
com:18080/api/v1/applications/application_1482090835601_0359/1/stages
com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "submissionTime" (class com.linkedin.drelephant.spark.fetchers.st
atusapiv1.StageData), not marked as ignorable (23 known properties: "inputBytes", "memoryBytesSpilled", "outputRecords", "numActiveTasks", "details",
"attemptId", "name", "tasks", "schedulingPool", "shuffleReadRecords", "executorSummary", "inputRecords", "diskBytesSpilled", "accumulatorUpdates", "ex
ecutorRunTime", "shuffleWriteBytes", "outputBytes", "numFailedTasks", "numCompleteTasks", "status", "stageId", "shuffleReadBytes", "shuffleWriteRecord
s"])
at [Source: [ {
"status" : "COMPLETE",
"stageId" : 1,
"attemptId" : 0,
"numActiveTasks" : 0,
"numCompleteTasks" : 122,
"numFailedTasks" : 0,
"executorRunTime" : 207244,
"submissionTime" : "2016-12-19T04:52:05.057GMT",
"firstTaskLaunchedTime" : "2016-12-19T04:52:05.058GMT",
"completionTime" : "2016-12-19T04:54:43.396GMT",
"inputBytes" : 0,
"inputRecords" : 0,
"outputBytes" : 0,
"outputRecords" : 0,
"shuffleReadBytes" : 2468364569,
"shuffleReadRecords" : 134378663,
"shuffleWriteBytes" : 0,
"shuffleWriteRecords" : 0,
"memoryBytesSpilled" : 0,
"diskBytesSpilled" : 0,
"name" : "collect at JavaWordCount.java:68",
"details" : "org.apache.spark.api.java.AbstractJavaRDDLike.collect(JavaRDDLike.scala:46)\norg.apache.spark.examples.JavaWordCount.main(JavaWordCount.java:68)\nsun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\nsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\nsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\njava.lang.reflect.Method.invoke(Method.java:498)\norg.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:542)",
"schedulingPool" : "default",
"accumulatorUpdates" : [ {
"id" : 2,
"name" : "peakExecutionMemory",
"value" : "302351628"
} ]
}
```

it also fails on `isActive` key.

```
12-20-2016 13:02:44 ERROR [ForkJoinPool-1-worker-5] com.linkedin.drelephant.spark.fetchers.SparkRestClient : error reading http://demo-nn1.company.
com:18080/api/v1/applications/application_1481846366268_2677/1/executors
com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "isActive" (class com.linkedin.drelephant.spark.fetchers.statusapiv1.ExecutorSummary), not marked as ignorable (15 known properties: "executorLogs", "completedTasks", "totalInputBytes", "maxMemory", "activeTasks", "hostPort", "totalTasks", "totalShuffleWrite", "totalShuffleRead", "failedTasks", "rddBlocks", "totalDuration", "diskUsed", "id", "memoryUsed"])
at [Source: [ {
"id" : "driver",
"hostPort" : "10.0.0.179:35425",
"isActive" : true,
"rddBlocks" : 0,
"memoryUsed" : 0,
"diskUsed" : 0,
"totalCores" : 0,
"maxTasks" : 0,
"activeTasks" : 0,
"failedTasks" : 0,
"completedTasks" : 0,
"totalTasks" : 0,
"totalDuration" : 0,
"totalGCTime" : 0,
"totalInputBytes" : 0,
"totalShuffleRead" : 0,
"totalShuffleWrite" : 0,
"maxMemory" : 2061647216,
"executorLogs" : {
"stderr" : "http://demo-dn4.company.com:8042/node/containerlogs/container_e21_1481846366268_2677_01_000001/alex/stderr?start=-4096",
"stdout" : "http://demo-dn4.company.com:8042/node/containerlogs/container_e21_1481846366268_2677_01_000001/alex/stdout?start=-4096"
}
}
```

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with SparkRestClient and the statusapiv1.StageData and ExecutorSummary classes named in the stack traces. Reproduce a Spark analysis using the shown stages and executors API responses, then inspect how those responses are deserialized. Done means both responses are accepted and the analysis no longer fails on submissionTime or isActive.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.