holdenk / holdenk/spark-testing-base

Tests always fail if I have more than one test file extends `StreamingSuiteBase`

Open
#341 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
1.6k
Forks
357
PR merge metrics
No merged PRs in 30d

Description

HI , thanks for the amazing library.

Here, I have a issue about using it , if I have more than one test, it always fail with error:
```
java.lang.IllegalStateException: Only one StreamingContext may be started in this JVM. Currently running StreamingContext was started atorg.apache.spark.streaming.StreamingContext.start(StreamingContext.scala:574)
[info] com.holdenkarau.spark.testing.StreamingSuiteCommon.withOutputAndStreamingContext(StreamingSuiteCommon.scala:121)
[info] com.holdenkarau.spark.testing.StreamingSuiteCommon.withOutputAndStreamingContext$(StreamingSuiteCommon.scala:114)
[info] net.truchat.sparkstreaming.AllStreamTest.withOutputAndStreamingContext(AllStreamTest.scala:7)
[info] com.holdenkarau.spark.testing.StreamingSuiteBase.testOperation(StreamingSuiteBase.scala:158)
[info] com.holdenkarau.spark.testing.StreamingSuiteBase.testOperation$(StreamingSuiteBase.scala:149)
```

If I put the two simple streaming tests in one test file with `extends FunSuite with StreamingSuiteBase`, it works.

My tests libs are as follows:
```
libraryDependencies += "org.scalactic" %% "scalactic" % "3.0.9",
libraryDependencies += "org.scalactic" %% "scalactic" % "3.0.9" % "test",
libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.9" % Test,
libraryDependencies += "com.holdenkarau" %% "spark-testing-base" % "3.1.2_1.1.0" % Test,
```

Scala version : `2.12.14`

Thanks

Contributor guide

Open the contributing guide

Research direction

Start with StreamingSuiteBase.testOperation and StreamingSuiteCommon.withOutputAndStreamingContext, then reproduce the failure using more than one test file extending StreamingSuiteBase. Compare that behavior with two tests in one file and determine what cleanup or isolation is missing; done means multiple test files can run without the StreamingContext exception.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.