DataflowRunner does not scale when reading gzip file
Open
bug
core
dataflow
P3
python
runners
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
Hi,
I have a pipe that ReadFromText() a 700mb gz file from a GS bucket.
It then parse json, create BigQuery row, and WriteToBigQuery.
The pipeline above does not scale. If I specify 2 workers on startup it will scale it down to 1 and the throughput remains the same. The job takes 30 minutes.
What I found is that the exact same pipeline, reading the same but uncompressed 11gb file from the same location scales very well. The job only takes 5 minutes.
Imported from Jira [BEAM-7094](https://issues.apache.org/jira/browse/BEAM-7094). Original Jira may contain additional context.
Reported by: moander2.
Contributor guide
Assessment
This issue has not been assessed yet.