linkedin / linkedin/dr-elephant
The percent of resources wasted was negative for yarn-cluster spark streaming job
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 1.4k
- Forks
- 839
- PR merge metrics
- No merged PRs in 30d
Description
12-29-2016 13:11:46 INFO [Thread-6] com.linkedin.drelephant.ElephantRunner : Job queue size is 1
12-29-2016 13:11:46 INFO [dr-el-executor-thread-2] com.linkedin.drelephant.ElephantRunner : Analyzing SPARK application_1482973897405_0006
12-29-2016 13:11:46 INFO [dr-el-executor-thread-2] com.linkedin.drelephant.spark.fetchers.SparkFetcher : Fetching data for application_1482973897405_0006
12-29-2016 13:11:46 INFO [ForkJoinPool-1-worker-7] com.linkedin.drelephant.spark.fetchers.SparkRestClient : calling REST API at http://{$ip}:18080/api/v1/applications/application_1482973897405_0006
12-29-2016 13:11:46 INFO [ForkJoinPool-1-worker-7] com.linkedin.drelephant.spark.fetchers.SparkLogClient : looking for logs at webhdfs://${ip}:50070/user/spark/eventLog/application_1482973897405_0006_1.snappy
12-29-2016 13:11:46 ERROR [dr-el-executor-thread-2] com.linkedin.drelephant.ElephantRunner : ERROR executing DML bindLog[] error[Data truncation: Out of range value for column 'resource_wasted' at row 1]
12-29-2016 13:11:46 ERROR [dr-el-executor-thread-2] com.linkedin.drelephant.ElephantRunner : javax.persistence.PersistenceException: ERROR executing DML bindLog[] error[Data truncation: Out of range value for column 'resource_wasted' at row 1]
at com.avaje.ebeaninternal.server.persist.dml.DmlBeanPersister.execute(DmlBeanPersister.java:97)
at com.avaje.ebeaninternal.server.persist.dml.DmlBeanPersister.insert(DmlBeanPersister.java:57)
at com.avaje.ebeaninternal.server.persist.DefaultPersistExecute.executeInsertBean(DefaultPersistExecute.java:66)
at com.avaje.ebeaninternal.server.core.PersistRequestBean.executeNow(PersistRequestBean.java:448)
at com.avaje.ebeaninternal.server.core.PersistRequestBean.executeOrQueue(PersistRequestBean.java:478)
at com.avaje.ebeaninternal.server.persist.DefaultPersister.insert(DefaultPersister.java:335)
at com.avaje.ebeaninternal.server.persist.DefaultPersister.saveEnhanced(DefaultPersister.java:310)
at com.avaje.ebeaninternal.server.persist.DefaultPersister.saveRecurse(DefaultPersister.java:280)
at com.avaje.ebeaninternal.server.persist.DefaultPersister.save(DefaultPersister.java:248)
at com.avaje.ebeaninternal.server.core.DefaultServer.save(DefaultServer.java:1568)
at com.avaje.ebeaninternal.server.core.DefaultServer.save(DefaultServer.java:1558)
at com.avaje.ebean.Ebean.save(Ebean.java:453)
at play.db.ebean.Model.save(Model.java:91)
at com.linkedin.drelephant.ElephantRunner$ExecutorJob.run(ElephantRunner.java:178)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.mysql.jdbc.MysqlDataTruncation: Data truncation: Out of range value for column 'resource_wasted' at row 1
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3845)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3783)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2447)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2594)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2545)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1901)
at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2113)
at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2049)
at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2034)
at com.jolbox.bonecp.PreparedStatementHandle.executeUpdate(PreparedStatementHandle.java:205)
at com.avaje.ebeaninternal.server.type.DataBind.executeUpdate(DataBind.java:55)
at com.avaje.ebeaninternal.server.persist.dml.InsertHandler.execute(InsertHandler.java:134)
at com.avaje.ebeaninternal.server.persist.dml.DmlBeanPersister.execute(DmlBeanPersister.java:86)
... 18 more
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the reported Spark streaming logs and the stack-trace entry point at ElephantRunner$ExecutorJob.run, then trace how resource_wasted is calculated and persisted for a yarn-cluster job. Reproduce the reported application if possible and verify that analysis completes without the MySQL out-of-range error and that the reported waste percentage is valid.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, mysql, spark
- Domain
- backend, databases, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100