Spark runner is not respecting spark.default.parallelism user defined configuration
Open
bug
P3
runners
spark
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
It was reported in [the mailing list](https://lists.apache.org/thread.html/792fb7fc2a5113837fbcdafce6a5d9100309881b366c1a7163d2c898@%3Cdev.beam.apache.org%3E) that the Spark runner is not respecting the user defined Spark default parallelism configuration. We should investigate and if it is the case ensure that a user defined configuration is always respected. Runner optimizations should apply only for default (unconfigured) values otherwise we will confuse users and limit them from parametrizing Spark for their best convenience.
Imported from Jira [BEAM-8384](https://issues.apache.org/jira/browse/BEAM-8384). Original Jira may contain additional context.
Reported by: iemejia.
Contributor guide
Assessment
This issue has not been assessed yet.