holdenk / holdenk/spark-testing-base

Exposing Builder object created in DataFrameSuiteBaseLike

Open
#259 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
1.6k
Forks
357
PR merge metrics
No merged PRs in 30d

Description

I'm in a situation where SQLContext fails to parse some CREATE TABLE statements that I call on test setup. But if I could talk directly to Hive (backed by MySQL), the SQL statements work.

So this forces me to set up an Hive instance instead of relying fully on SparkSql/Spark's-embedded-hive functionality.

I'm now starting MiniMRYarnCluster and HiveServer2 in the test code, and then want to make spark-testing-base use that embedded Hive server instead of the default Spark one.

I've got a POC working by extending `DataFrameSuiteBaseLike` with a custom trait and putting it in the same package `com.holdenkarau.spark.testing` (which I had to do because `WrappedConfVar` is private).

So with all that said, I think the right thing to do at this point is for me to make a PR which gives users of this library a chance to change the SparkSession Builder object created in `DataFrameSuiteBaseLike.newBuilder` with their own code.

Thoughts?

Contributor guide

Open the contributing guide

Research direction

Start by reading DataFrameSuiteBaseLike.newBuilder and the surrounding WrappedConfVar access in the com.holdenkarau.spark.testing package. Review the issue's custom-trait POC and existing test setup to determine how a caller should provide a custom SparkSession Builder. Done means users can replace or modify that builder without extending the trait or relying on package-private access.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
testing
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.