linkedin / linkedin/dr-elephant
Dr.Elephant and spark 2.1
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 1.4k
- Forks
- 839
- PR merge metrics
- No merged PRs in 30d
Description
Not loading spark jobs and throwing errors in dr_elephant.log. Any suggestions? Thanks.
ERROR [dr-el-executor-thread-2] com.linkedin.drelephant.spark.fetchers.SparkFetcher : Failed fetching data for application_1496678777212_0026
com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "lastUpdated" (class com.linkedin.drelephant.spark.fetchers.statusapiv1.ApplicationAttemptInfo), not marked as ignorable (5 known properties: "attemptId", "sparkUser", "endTime", "completed", "startTime"])
(through reference chain: com.fasterxml.jackson.module.scala.deser.BuilderWrapper[0]->com.linkedin.drelephant.spark.fetchers.statusapiv1.ApplicationAttemptInfo["lastUpdated"])
at com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:51)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the dr_elephant.log stack trace and inspect SparkFetcher and ApplicationAttemptInfo, where the reported lastUpdated field is rejected. Compare the expected application status data for Spark 2.1 with the fields handled there, and verify that Spark jobs load without the reported exception.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, java
- Domain
- data-engineering, observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100