[Bug]: No parallelism using WriteToParquet in Apache Spark
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
### What happened?
When running Beam on Spark using `WriteToParquet` without `num_shards`, files seem to be written with no parallelism. In https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.io.parquetio.html it says:
> num_shards – The number of files (shards) used for output. If not set, the service will decide on the optimal number of shards.
However, in Spark, my tasks looks like this:

I believe that this is happening because `iobase.WriteImpl` in [here](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/iobase.py#L1156-L1157) is doing:
```
...
| 'Pair' >> core.Map(lambda x: (None, x))
| core.GroupByKey()
```
which was added in this PR: https://github.com/apache/beam/pull/958
If I understand correctly, the pcollection elements will all have the same key, `None`, and `GroupByKey` will group all those elements into a single "partition" (in Spark terms). This "None" partition is massively skewed and can only be written by 1 thread / task and will take forever.
### Issue Priority
Priority: 2
### Issue Component
Component: io-py-parquet
Contributor guide
Assessment
This issue has not been assessed yet.