linkedin / linkedin/dr-elephant
Does not compile with Spark 2.0.0
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 1.4k
- Forks
- 839
- PR merge metrics
- No merged PRs in 30d
Description
Hi,
I am trying to compile with Spark 2.0.0, Hadoop 2.7.0
and getting the following issue:
[info] 'compiler-interface' not yet compiled for Scala 2.10.4. Compiling...
[info] Compilation completed in 7.003 s
[error] /../dr-elephant-master/app/org/apache/spark/deploy/history/SparkDataCollection.scala:52: not enough arguments for constructor StorageStatusListener: (conf: org.apache.spark.SparkConf)org.apache.spark.storage.StorageStatusListener.
[error] Unspecified value parameter conf.
[error] lazy val storageStatusListener = new StorageStatusListener()
[error] ^
[error] /../dr-elephant-master/app/org/apache/spark/deploy/history/SparkDataCollection.scala:53: not enough arguments for constructor ExecutorsListener: (storageStatusListener: org.apache.spark.storage.StorageStatusListener, conf: org.apache.spark.SparkConf)org.apache.spark.ui.exec.ExecutorsListener.
[error] Unspecified value parameter conf.
[error] lazy val executorsListener = new ExecutorsListener(storageStatusListener)
[error] ^
[warn] /../dr-elephant-master/app/org/apache/spark/deploy/history/SparkDataCollection.scala:313: abstract type pattern T is unchecked since it is eliminated by erasure
[warn] seq.foreach { case (item: T) => list.add(item)}
[warn] ^
[error] /../dr-elephant-master/app/org/apache/spark/storage/StorageStatusTrackingListener.scala:80: value updatedBlocks is not a member of org.apache.spark.executor.TaskMetrics
[error] val updatedBlocks = metrics.updatedBlocks.getOrElse(Seq[(BlockId, BlockStatus)]())
[error]
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with app/org/apache/spark/deploy/history/SparkDataCollection.scala and app/org/apache/spark/storage/StorageStatusTrackingListener.scala, then compare the referenced Spark listener and TaskMetrics APIs with Spark 2.0.0. Run the project compilation against Spark 2.0.0 and Hadoop 2.7.0; done means the reported constructor and updatedBlocks errors are resolved and compilation succeeds.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, scala, spark
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100