holdenk/spark-testing-base

View on GitHub

Base classes to use when writing tests with Spark

Stars: 1.6k
Forks: 357
Open beginner issues: 0
Indexed issues: 80

Dominant language: Scala
License: Apache-2.0
Last GitHub push: Aug 7, 2026
Latest indexed: Sep 17, 2026
Contributing guide: Contributing guide
Code of conduct: Code of conduct
Beginner labels: No beginner labels indexed
PR merge metrics: No merged PRs in 30d

Open All

80 open issues indexed

Add support for quickcheck DStreams Open

holdenk/spark-testing-base#15 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Add Hypothesis for Python Open

holdenk/spark-testing-base#23 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
More documentation! Open

holdenk/spark-testing-base#37 · 8 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Add a code of conduct Open

holdenk/spark-testing-base#47 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Create project page at github.io Open

holdenk/spark-testing-base#58 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
how to test foreachRDD in python Open

holdenk/spark-testing-base#63 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Nice to have more types in DataFrameSuiteBaseLike.approxEquals Open

holdenk/spark-testing-base#73 · 4 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Run Streaming Tests Distributively Open

holdenk/spark-testing-base#75 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Not allowed to query with all of partition Keys(cassandra) with Equals Open

holdenk/spark-testing-base#101 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Is it possible to use DataFrameSuiteBase with FunSpec(scalatest)? Open

holdenk/spark-testing-base#108 · 6 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Add test for TimeStamp generator Open

holdenk/spark-testing-base#115 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
JavaRDDComparisons on RDDs with Tuples with Decimals Open

holdenk/spark-testing-base#117 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Created shared version of DataFrameSuiteBase Open

holdenk/spark-testing-base#123 · 0 comments · 3 reactions · 0 assignees · Indexed Sep 17, 2026
Switch hive dependency to non-test dep and add a note for people that want to exclude it. Open

holdenk/spark-testing-base#124 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Support for ScalaTest maven plugin. Open

holdenk/spark-testing-base#128 · 0 comments · 5 reactions · 0 assignees · Indexed Sep 17, 2026
specs2 support ? Open

holdenk/spark-testing-base#134 · 3 comments · 1 reaction · 0 assignees · Indexed Sep 17, 2026
Checkpoint directory has not been set in the SparkContext Open

holdenk/spark-testing-base#136 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 17, 2026
StreamingSuitBase test method ERROR Open

holdenk/spark-testing-base#142 · 8 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Plan to decouple with UnitTest? Open

holdenk/spark-testing-base#161 · 3 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
java.lang.IncompatibleClassChangeError Open

holdenk/spark-testing-base#163 · 6 comments · 1 reaction · 0 assignees · Indexed Sep 17, 2026
Testing Structured Streaming Use Cases Open

holdenk/spark-testing-base#164 · 4 comments · 5 reactions · 0 assignees · Indexed Sep 17, 2026
Error while instantiating 'org.apache.spark.sql.hive.HiveSessionState' on DataFrameSuiteBase Open

holdenk/spark-testing-base#167 · 8 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Streaming testOperation() for more than 2 streams Open

holdenk/spark-testing-base#168 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Minicluster throws NPE on shutdown Open

holdenk/spark-testing-base#171 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 17, 2026
DataFrame comparision with different schema metadata error message is confusing Open

holdenk/spark-testing-base#179 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 17, 2026
NPE when using SparkSession from DataFrameSuiteBase Open

holdenk/spark-testing-base#180 · 6 comments · 2 reactions · 0 assignees · Indexed Sep 17, 2026
Remove EvilSessionTools / SparkSession from DataFrameSuiteBase Open

holdenk/spark-testing-base#186 · 2 comments · 1 reaction · 0 assignees · Indexed Sep 17, 2026
Just make hive explicitly required Open

holdenk/spark-testing-base#198 · 0 comments · 3 reactions · 0 assignees · Indexed Sep 17, 2026
DataframeGenerator : Ensure minimum number of distinct values Open

holdenk/spark-testing-base#211 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
approxEquals with relative tolerance Open

holdenk/spark-testing-base#214 · 3 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
DataframeSuiteBase approxEquals(): need support with specified tolerence type Open

holdenk/spark-testing-base#221 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
python - unnecessary dependency on pytest in setup.py? Open

holdenk/spark-testing-base#226 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Length & schema checks report expected as actual Open

holdenk/spark-testing-base#230 · 4 comments · 1 reaction · 0 assignees · Indexed Sep 17, 2026
Hive support lost when upgrading from spark-testing-base 2.2.0_0.9.0 to 2.3.0_0.9.0 Open

holdenk/spark-testing-base#234 · 15 comments · 3 reactions · 0 assignees · Indexed Sep 17, 2026
PySpark: SparkTestingBaseTestCase.assertRDDEquals is not an assertion Open

holdenk/spark-testing-base#235 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 17, 2026
assertDataFrameEquals message says that 2 schemas are different after printing out the exact same schemas Open

holdenk/spark-testing-base#236 · 4 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
RDDComparisons could use a more descriptive error Open

holdenk/spark-testing-base#237 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Multiple Coverage Highlighting in the build Open

holdenk/spark-testing-base#239 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Parallel Execution Open

holdenk/spark-testing-base#240 · 6 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
SqlContext test cases not running.. Open

holdenk/spark-testing-base#242 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
assertDataFrameApproximateEquals failed Open

holdenk/spark-testing-base#243 · 0 comments · 1 reaction · 0 assignees · Indexed Sep 17, 2026
Comparing RDD of org.apache.spark.sql.Row containing byte[] as values fails for the same provided RDD Open

holdenk/spark-testing-base#247 · 5 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Exposing Builder object created in DataFrameSuiteBaseLike Open

holdenk/spark-testing-base#259 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
hadoop-hdfs: NoSuchMethodError - org.apache.hadoop.tracing.SpanReceiverHost.getInstance Open

holdenk/spark-testing-base#260 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Global spark configuration changes are not maintained during Maven tests Open

holdenk/spark-testing-base#261 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Temporary warehouse on local is not a valid hdfs path (SharedSparkContext) Open

holdenk/spark-testing-base#267 · 0 comments · 1 reaction · 0 assignees · Indexed Sep 17, 2026
Misleading error when comparing resutls Open

holdenk/spark-testing-base#269 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
More info when comparing different DataFrame/DataSet Open

holdenk/spark-testing-base#272 · 3 comments · 9 reactions · 0 assignees · Indexed Sep 17, 2026
Use same Hadoop version as spark target? Open

holdenk/spark-testing-base#278 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 17, 2026
Explore if pytest-xdist can work with spark tests Open

holdenk/spark-testing-base#279 · 1 comment · 1 reaction · 0 assignees · Indexed Sep 17, 2026

holdenk/spark-testing-base

Get new issues in your inbox