holdenk / holdenk/spark-testing-base
Global spark configuration changes are not maintained during Maven tests
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 357
- PR merge metrics
- No merged PRs in 30d
Description
Related to this issue: https://github.com/holdenk/spark-testing-base/issues/187
I want to increase spark.driver.maxResultSize; not just for testing but globally. I have it set in `conf/spark-default.conf` as
`spark.driver.maxResultSize 2g`
When I start a spark-shell or spark-submit a job; I can see that this is reflected in my Environment variables at http://:4040. However, when I run a `mvn test` using spark-testing-base, I get
```
[Stage 25:> (0 + 4) / 4]
18/08/14 17:24:16 ERROR TaskSetManager: Total size of serialized results of 3 tasks (1150.1 MB) is bigger than spark.driver.maxResultSize (1024.0 MB)
18/08/14 17:24:16 ERROR TaskSetManager: Total size of serialized results of 4 tasks (1533.5 MB) is bigger than spark.driver.maxResultSize (1024.0 MB)
```
This is the normal default spark.driver.maxResultSize. I'm can't determine if spark-testing-base is supposed to be using the configuration settings at `conf/spark-default.conf` or not, so I don't know if this is expected behavior or not. I don't see anywhere where a SparkConf object would override my conf file.
This is related to the post linked at the top, where you have two solutions for the problem I have. But, I'd like to know if changing the `conf/spark-default.conf` is another valid solution or not.
Contributor guide
Research direction
Start with conf/spark-default.conf and the spark-testing-base behavior exercised by mvn test; compare those settings with the reported spark-shell and spark-submit environment. Review the related issue #187 for context, then determine whether the configured spark.driver.maxResultSize should be honored during tests and document or verify the expected behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- build-system, testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100