linkedin / linkedin/dr-elephant

The percent of resources wasted was negative for yarn-cluster spark streaming job

Open
#180 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
1.4k
Forks
839
PR merge metrics
No merged PRs in 30d

Description

12-29-2016 13:11:46 INFO [Thread-6] com.linkedin.drelephant.ElephantRunner : Job queue size is 1
12-29-2016 13:11:46 INFO [dr-el-executor-thread-2] com.linkedin.drelephant.ElephantRunner : Analyzing SPARK application_1482973897405_0006
12-29-2016 13:11:46 INFO [dr-el-executor-thread-2] com.linkedin.drelephant.spark.fetchers.SparkFetcher : Fetching data for application_1482973897405_0006
12-29-2016 13:11:46 INFO [ForkJoinPool-1-worker-7] com.linkedin.drelephant.spark.fetchers.SparkRestClient : calling REST API at http://{$ip}:18080/api/v1/applications/application_1482973897405_0006
12-29-2016 13:11:46 INFO [ForkJoinPool-1-worker-7] com.linkedin.drelephant.spark.fetchers.SparkLogClient : looking for logs at webhdfs://${ip}:50070/user/spark/eventLog/application_1482973897405_0006_1.snappy
12-29-2016 13:11:46 ERROR [dr-el-executor-thread-2] com.linkedin.drelephant.ElephantRunner : ERROR executing DML bindLog[] error[Data truncation: Out of range value for column 'resource_wasted' at row 1]
12-29-2016 13:11:46 ERROR [dr-el-executor-thread-2] com.linkedin.drelephant.ElephantRunner : javax.persistence.PersistenceException: ERROR executing DML bindLog[] error[Data truncation: Out of range value for column 'resource_wasted' at row 1]
at com.avaje.ebeaninternal.server.persist.dml.DmlBeanPersister.execute(DmlBeanPersister.java:97)
at com.avaje.ebeaninternal.server.persist.dml.DmlBeanPersister.insert(DmlBeanPersister.java:57)
at com.avaje.ebeaninternal.server.persist.DefaultPersistExecute.executeInsertBean(DefaultPersistExecute.java:66)
at com.avaje.ebeaninternal.server.core.PersistRequestBean.executeNow(PersistRequestBean.java:448)
at com.avaje.ebeaninternal.server.core.PersistRequestBean.executeOrQueue(PersistRequestBean.java:478)
at com.avaje.ebeaninternal.server.persist.DefaultPersister.insert(DefaultPersister.java:335)
at com.avaje.ebeaninternal.server.persist.DefaultPersister.saveEnhanced(DefaultPersister.java:310)
at com.avaje.ebeaninternal.server.persist.DefaultPersister.saveRecurse(DefaultPersister.java:280)
at com.avaje.ebeaninternal.server.persist.DefaultPersister.save(DefaultPersister.java:248)
at com.avaje.ebeaninternal.server.core.DefaultServer.save(DefaultServer.java:1568)
at com.avaje.ebeaninternal.server.core.DefaultServer.save(DefaultServer.java:1558)
at com.avaje.ebean.Ebean.save(Ebean.java:453)
at play.db.ebean.Model.save(Model.java:91)
at com.linkedin.drelephant.ElephantRunner$ExecutorJob.run(ElephantRunner.java:178)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.mysql.jdbc.MysqlDataTruncation: Data truncation: Out of range value for column 'resource_wasted' at row 1
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3845)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3783)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2447)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2594)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2545)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1901)
at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2113)
at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2049)
at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2034)
at com.jolbox.bonecp.PreparedStatementHandle.executeUpdate(PreparedStatementHandle.java:205)
at com.avaje.ebeaninternal.server.type.DataBind.executeUpdate(DataBind.java:55)
at com.avaje.ebeaninternal.server.persist.dml.InsertHandler.execute(InsertHandler.java:134)
at com.avaje.ebeaninternal.server.persist.dml.DmlBeanPersister.execute(DmlBeanPersister.java:86)
... 18 more

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the reported Spark streaming logs and the stack-trace entry point at ElephantRunner$ExecutorJob.run, then trace how resource_wasted is calculated and persisted for a yarn-cluster job. Reproduce the reported application if possible and verify that analysis completes without the MySQL out-of-range error and that the reported waste percentage is valid.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mysql, spark
Domain
backend, databases, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.