holdenk / holdenk/spark-testing-base
Checkpoint directory has not been set in the SparkContext
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 357
- PR merge metrics
- No merged PRs in 30d
Description
I get this exception when I use spark-testing-base
```
Checkpoint directory has not been set in the SparkContext
org.apache.spark.SparkException: Checkpoint directory has not been set in the SparkContext
at org.apache.spark.rdd.RDD.checkpoint(RDD.scala:1495)
at pl.appsilon.marek.sparkdatalog.eval.DatabaseRepr$$anonfun$checkpoint$1.apply(DatabaseRepr.scala:13)
at pl.appsilon.marek.sparkdatalog.eval.DatabaseRepr$$anonfun$checkpoint$1.apply(DatabaseRepr.scala:13)
at scala.collection.MapLike$FilteredKeys$$anonfun$foreach$1.apply(MapLike.scala:231)
at scala.collection.MapLike$FilteredKeys$$anonfun$foreach$1.apply(MapLike.scala:231)
at scala.collection.immutable.Map$Map2.foreach(Map.scala:137)
at scala.collection.MapLike$FilteredKeys.foreach(MapLike.scala:231)
```
I am using this version
```
libraryDependencies += "com.holdenkarau" % "spark-testing-base_2.11" % "2.0.0_0.4.5"
```
Contributor guide
Research direction
Reproduce the exception with spark-testing-base version 2.0.0_0.4.5 and the shown dependency. Start from the SparkContext setup and the stack trace entry at DatabaseRepr.scala:13, then determine how checkpointing is expected to be configured; done means the reported checkpoint exception is resolved and covered by a test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100