linkedin / linkedin/dr-elephant
Exception parsing Spark event log
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 1.4k
- Forks
- 839
- PR merge metrics
- No merged PRs in 30d
Description
Two recent compiles, one using sbt.version 0.13.9 and the other using 0.13.2 in build.properties.
Compiled on centos7 with activator 1.3.12
Cluster is ambari hdp stack 2.6.1
Both have the below values in compile.conf
hadoop_version=2.6.1
spark_version=2.1.1
play_opts="-Dsbt.repository.config=app-conf/resolver.conf"
These are two separate compilations on two separate boxes that are members of the same Ambari cluster structure (hdp stack 2.6.1) and both hosts fire the error below. Note that I am only executing included spark test jobs (word count of war-and-peace and sparkpi)
`[error] o.a.s.s.ReplayListenerBus - Exception parsing Spark event log: application_1502904041499_0358
org.json4s.package$MappingException: Did not find value which can be converted into boolean
at org.json4s.reflect.package$.fail(package.scala:96) ~[org.json4s.json4s-core_2.10-3.2.10.jar:3.2.10]
at org.json4s.Extraction$.convert(Extraction.scala:554) ~[org.json4s.json4s-core_2.10-3.2.10.jar:3.2.10]
at org.json4s.Extraction$.extract(Extraction.scala:331) ~[org.json4s.json4s-core_2.10-3.2.10.jar:3.2.10]
at org.json4s.Extraction$.extract(Extraction.scala:42) ~[org.json4s.json4s-core_2.10-3.2.10.jar:3.2.10]
at org.json4s.ExtractableJsonAstNode.extract(ExtractableJsonAstNode.scala:21) ~[org.json4s.json4s-core_2.10-3.2.10.jar:3.2.10]
at org.apache.spark.util.JsonProtocol$.storageLevelFromJson(JsonProtocol.scala:826) ~[org.apache.spark.spark-core_2.10-1.4.0.jar:1.4.0]
[error] o.a.s.s.ReplayListenerBus - Malformed line #21: {"Event":"SparkListenerJobStart","Job ID":0,"Submission Time":1503944198210,"Stage Infos":[{"Stage ID":0,"Stage Attempt ID":0,"Stage Name":"reduce at SparkPi.scala:38","Number of Tasks":100000,"RDD Info":[{"RDD ID":1,"Name":"MapPartitionsRDD","Scope":"{\"id\":\"1\",\"name\":\"map\"}","Callsite":"map at SparkPi.scala:34","Parent IDs":[0],"Storage Level":{"Use Disk":false,"Use Memory":false,"Deserialized":false,"Replication":1},"Number of Partitions":100000,"Number of Cached Partitions":0,"Memory Size":0,"Disk Size":0},{"RDD ID":0,"Name":"ParallelCollectionRDD","Scope":"{\"id\":\"0\",\"name\":\"parallelize\"}","Callsite":"parallelize at SparkPi.scala:34","Parent IDs":[],"Storage Level":{"Use Disk":false,"Use Memory":false,"Deserialized":false,"Replication":1},"Number of Partitions":100000,"Number of Cached Partitions":0,"Memory Size":0,"Disk Size":0}],"Parent IDs":[],"Details":"org.apache.spark.rdd.RDD.reduce(RDD.scala:1008)\norg.apache.spark.examples.SparkPi$.main(SparkPi.scala:38)\norg.apache.spark.examples.SparkPi.main(SparkPi.scala)\nsun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\nsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\nsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\njava.lang.reflect.Method.invoke(Method.java:498)\norg.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:750)\norg.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)\norg.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)\norg.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)\norg.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)","Accumulables":[]}],"Stage IDs":[0],"Properties":{"spark.rdd.scope.noOverride":"true","spark.rdd.scope":"{\"id\":\"2\",\"name\":\"reduce\"}"}}
`
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The report points to Spark's ReplayListenerBus and JsonProtocol.storageLevelFromJson, and includes the malformed SparkListenerJobStart line; no Dr. Elephant source file or test is named. Start by tracing how Spark event logs are replayed, then reproduce with the included SparkPi or word-count jobs; done means the reported event log parses without the boolean conversion error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, spark
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100