holdenk / holdenk/spark-testing-base

Global spark configuration changes are not maintained during Maven tests

Open
#261 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
1.6k
Forks
357
PR merge metrics
No merged PRs in 30d

Description

Related to this issue: https://github.com/holdenk/spark-testing-base/issues/187

I want to increase spark.driver.maxResultSize; not just for testing but globally. I have it set in `conf/spark-default.conf` as

`spark.driver.maxResultSize 2g`

When I start a spark-shell or spark-submit a job; I can see that this is reflected in my Environment variables at http://:4040. However, when I run a `mvn test` using spark-testing-base, I get

```
[Stage 25:> (0 + 4) / 4]
18/08/14 17:24:16 ERROR TaskSetManager: Total size of serialized results of 3 tasks (1150.1 MB) is bigger than spark.driver.maxResultSize (1024.0 MB)
18/08/14 17:24:16 ERROR TaskSetManager: Total size of serialized results of 4 tasks (1533.5 MB) is bigger than spark.driver.maxResultSize (1024.0 MB)
```

This is the normal default spark.driver.maxResultSize. I'm can't determine if spark-testing-base is supposed to be using the configuration settings at `conf/spark-default.conf` or not, so I don't know if this is expected behavior or not. I don't see anywhere where a SparkConf object would override my conf file.

This is related to the post linked at the top, where you have two solutions for the problem I have. But, I'd like to know if changing the `conf/spark-default.conf` is another valid solution or not.

Contributor guide

Open the contributing guide

Research direction

Start with conf/spark-default.conf and the spark-testing-base behavior exercised by mvn test; compare those settings with the reported spark-shell and spark-submit environment. Review the related issue #187 for context, then determine whether the configured spark.driver.maxResultSize should be honored during tests and document or verify the expected behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
build-system, testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.