Setting dedicated warehouse and metastore temp-directory doens't work

Hello,
I've noticed, when I run one of my tests, the default warehouse directory (current directory is used. Same for metastore directory. This directories will be reused by each Spark Session, therefore, my tests will have dependencies between each other.

I've found out that the framework actually handles that problem by setting these directories, among other configuration. See

DataFrameSuiteBaseLike#sqlBeforeAllTestCases

However, these settings are never used, because at the moment you're creating this configuration, the Spark Context will already be created. And therefore all future settings are ignored. More precisely, the SharedSparkContext#beforeAll method is called before DataFrameSuiteBaseLike#sqlBeforeAllTestCases

Unfortunately, I haven't found a way to fix it. Therefore, I won't be able to make a PR. Maybe someone else might help out.

Thanks.
Kuno

Trace DataFrameSuiteBaseLike#sqlBeforeAllTestCases and SharedSparkContext#beforeAll to confirm the Spark Context is created before the warehouse and metastore directories are configured. Reproduce the issue with the affected test setup, then verify that each Spark Session uses dedicated directories and tests no longer depend on one another.

Written by the indexing model from the issue text.