linkedin / linkedin/dr-elephant
Elephant Runner fails on SPARK analysis with new Fetcher/Metrics/Heuristics
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 1.4k
- Forks
- 839
- PR merge metrics
- No merged PRs in 30d
Description
```
12-20-2016 13:15:31 ERROR [ForkJoinPool-1-worker-1] com.linkedin.drelephant.spark.fetchers.SparkRestClient : error reading http://demo-nn1.company.
com:18080/api/v1/applications/application_1482090835601_0359/1/stages
com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "submissionTime" (class com.linkedin.drelephant.spark.fetchers.st
atusapiv1.StageData), not marked as ignorable (23 known properties: "inputBytes", "memoryBytesSpilled", "outputRecords", "numActiveTasks", "details",
"attemptId", "name", "tasks", "schedulingPool", "shuffleReadRecords", "executorSummary", "inputRecords", "diskBytesSpilled", "accumulatorUpdates", "ex
ecutorRunTime", "shuffleWriteBytes", "outputBytes", "numFailedTasks", "numCompleteTasks", "status", "stageId", "shuffleReadBytes", "shuffleWriteRecord
s"])
at [Source: [ {
"status" : "COMPLETE",
"stageId" : 1,
"attemptId" : 0,
"numActiveTasks" : 0,
"numCompleteTasks" : 122,
"numFailedTasks" : 0,
"executorRunTime" : 207244,
"submissionTime" : "2016-12-19T04:52:05.057GMT",
"firstTaskLaunchedTime" : "2016-12-19T04:52:05.058GMT",
"completionTime" : "2016-12-19T04:54:43.396GMT",
"inputBytes" : 0,
"inputRecords" : 0,
"outputBytes" : 0,
"outputRecords" : 0,
"shuffleReadBytes" : 2468364569,
"shuffleReadRecords" : 134378663,
"shuffleWriteBytes" : 0,
"shuffleWriteRecords" : 0,
"memoryBytesSpilled" : 0,
"diskBytesSpilled" : 0,
"name" : "collect at JavaWordCount.java:68",
"details" : "org.apache.spark.api.java.AbstractJavaRDDLike.collect(JavaRDDLike.scala:46)\norg.apache.spark.examples.JavaWordCount.main(JavaWordCount.java:68)\nsun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\nsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\nsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\njava.lang.reflect.Method.invoke(Method.java:498)\norg.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:542)",
"schedulingPool" : "default",
"accumulatorUpdates" : [ {
"id" : 2,
"name" : "peakExecutionMemory",
"value" : "302351628"
} ]
}
```
it also fails on `isActive` key.
```
12-20-2016 13:02:44 ERROR [ForkJoinPool-1-worker-5] com.linkedin.drelephant.spark.fetchers.SparkRestClient : error reading http://demo-nn1.company.
com:18080/api/v1/applications/application_1481846366268_2677/1/executors
com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "isActive" (class com.linkedin.drelephant.spark.fetchers.statusapiv1.ExecutorSummary), not marked as ignorable (15 known properties: "executorLogs", "completedTasks", "totalInputBytes", "maxMemory", "activeTasks", "hostPort", "totalTasks", "totalShuffleWrite", "totalShuffleRead", "failedTasks", "rddBlocks", "totalDuration", "diskUsed", "id", "memoryUsed"])
at [Source: [ {
"id" : "driver",
"hostPort" : "10.0.0.179:35425",
"isActive" : true,
"rddBlocks" : 0,
"memoryUsed" : 0,
"diskUsed" : 0,
"totalCores" : 0,
"maxTasks" : 0,
"activeTasks" : 0,
"failedTasks" : 0,
"completedTasks" : 0,
"totalTasks" : 0,
"totalDuration" : 0,
"totalGCTime" : 0,
"totalInputBytes" : 0,
"totalShuffleRead" : 0,
"totalShuffleWrite" : 0,
"maxMemory" : 2061647216,
"executorLogs" : {
"stderr" : "http://demo-dn4.company.com:8042/node/containerlogs/container_e21_1481846366268_2677_01_000001/alex/stderr?start=-4096",
"stdout" : "http://demo-dn4.company.com:8042/node/containerlogs/container_e21_1481846366268_2677_01_000001/alex/stdout?start=-4096"
}
}
```
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with SparkRestClient and the statusapiv1.StageData and ExecutorSummary classes named in the stack traces. Reproduce a Spark analysis using the shown stages and executors API responses, then inspect how those responses are deserialized. Done means both responses are accepted and the analysis no longer fails on submissionTime or isActive.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100