holdenk / holdenk/spark-testing-base
Exposing Builder object created in DataFrameSuiteBaseLike
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 357
- PR merge metrics
- No merged PRs in 30d
Description
I'm in a situation where SQLContext fails to parse some CREATE TABLE statements that I call on test setup. But if I could talk directly to Hive (backed by MySQL), the SQL statements work.
So this forces me to set up an Hive instance instead of relying fully on SparkSql/Spark's-embedded-hive functionality.
I'm now starting MiniMRYarnCluster and HiveServer2 in the test code, and then want to make spark-testing-base use that embedded Hive server instead of the default Spark one.
I've got a POC working by extending `DataFrameSuiteBaseLike` with a custom trait and putting it in the same package `com.holdenkarau.spark.testing` (which I had to do because `WrappedConfVar` is private).
So with all that said, I think the right thing to do at this point is for me to make a PR which gives users of this library a chance to change the SparkSession Builder object created in `DataFrameSuiteBaseLike.newBuilder` with their own code.
Thoughts?
Contributor guide
Research direction
Start by reading DataFrameSuiteBaseLike.newBuilder and the surrounding WrappedConfVar access in the com.holdenkarau.spark.testing package. Review the issue's custom-trait POC and existing test setup to determine how a caller should provide a custom SparkSession Builder. Done means users can replace or modify that builder without extending the trait or relying on package-private access.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- testing
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100