holdenk / holdenk/spark-testing-base

Checkpoint directory has not been set in the SparkContext

Open
#136 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
1.6k
Forks
357
PR merge metrics
No merged PRs in 30d

Description

I get this exception when I use spark-testing-base

```
Checkpoint directory has not been set in the SparkContext
org.apache.spark.SparkException: Checkpoint directory has not been set in the SparkContext
at org.apache.spark.rdd.RDD.checkpoint(RDD.scala:1495)
at pl.appsilon.marek.sparkdatalog.eval.DatabaseRepr$$anonfun$checkpoint$1.apply(DatabaseRepr.scala:13)
at pl.appsilon.marek.sparkdatalog.eval.DatabaseRepr$$anonfun$checkpoint$1.apply(DatabaseRepr.scala:13)
at scala.collection.MapLike$FilteredKeys$$anonfun$foreach$1.apply(MapLike.scala:231)
at scala.collection.MapLike$FilteredKeys$$anonfun$foreach$1.apply(MapLike.scala:231)
at scala.collection.immutable.Map$Map2.foreach(Map.scala:137)
at scala.collection.MapLike$FilteredKeys.foreach(MapLike.scala:231)
```

I am using this version

```
libraryDependencies += "com.holdenkarau" % "spark-testing-base_2.11" % "2.0.0_0.4.5"

```

Contributor guide

Open the contributing guide

Research direction

Reproduce the exception with spark-testing-base version 2.0.0_0.4.5 and the shown dependency. Start from the SparkContext setup and the stack trace entry at DatabaseRepr.scala:13, then determine how checkpointing is expected to be configured; done means the reported checkpoint exception is resolved and covered by a test.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.