holdenk / holdenk/spark-testing-base
Tests always fail if I have more than one test file extends `StreamingSuiteBase`
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 357
- PR merge metrics
- No merged PRs in 30d
Description
HI , thanks for the amazing library.
Here, I have a issue about using it , if I have more than one test, it always fail with error:
```
java.lang.IllegalStateException: Only one StreamingContext may be started in this JVM. Currently running StreamingContext was started atorg.apache.spark.streaming.StreamingContext.start(StreamingContext.scala:574)
[info] com.holdenkarau.spark.testing.StreamingSuiteCommon.withOutputAndStreamingContext(StreamingSuiteCommon.scala:121)
[info] com.holdenkarau.spark.testing.StreamingSuiteCommon.withOutputAndStreamingContext$(StreamingSuiteCommon.scala:114)
[info] net.truchat.sparkstreaming.AllStreamTest.withOutputAndStreamingContext(AllStreamTest.scala:7)
[info] com.holdenkarau.spark.testing.StreamingSuiteBase.testOperation(StreamingSuiteBase.scala:158)
[info] com.holdenkarau.spark.testing.StreamingSuiteBase.testOperation$(StreamingSuiteBase.scala:149)
```
If I put the two simple streaming tests in one test file with `extends FunSuite with StreamingSuiteBase`, it works.
My tests libs are as follows:
```
libraryDependencies += "org.scalactic" %% "scalactic" % "3.0.9",
libraryDependencies += "org.scalactic" %% "scalactic" % "3.0.9" % "test",
libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.9" % Test,
libraryDependencies += "com.holdenkarau" %% "spark-testing-base" % "3.1.2_1.1.0" % Test,
```
Scala version : `2.12.14`
Thanks
Contributor guide
Research direction
Start with StreamingSuiteBase.testOperation and StreamingSuiteCommon.withOutputAndStreamingContext, then reproduce the failure using more than one test file extending StreamingSuiteBase. Compare that behavior with two tests in one file and determine what cleanup or isolation is missing; done means multiple test files can run without the StreamingContext exception.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100