Python BigQuery performance much worse than Java

The performance of reading from BigQuery in Python seems to be much worse than the performance of it in Java.

To reproduce this, I've run the following two programs on the Google Cloud, which basically read the weights from the public data set "natality" and outputs the top 100 largest weights.

Python:
```

#

options = PipelineOptions()
options.view_as(StandardOptions).runner = 'DataflowRunner'
#

pipeline = Pipeline(options=options)
(pipeline
| 'Read' >> beam.io.Read(beam.io.BigQuerySource(query='SELECT
weight_pounds FROM [bigquery-public-data:samples.natality]'))
| 'MapToFloat' >> beam.Map(lambda
elem: elem['weight_pounds'])
| 'Top' >> beam.combiners.Top.Largest(100)
| 'MapToString' >>
beam.Map(lambda elem: str(elem))
| 'Write' >> beam.io.WriteToText(""))

pipeline.run()

```

 Java:
```

//

public class Natality {
public static void main(String[] args) {

DataflowPipelineOptions options = PipelineOptionsFactory.create().as(DataflowPipelineOptions.class);

options.setRunner(DataflowRunner.class);
//

Pipeline
pipeline = Pipeline.create(options);

pipeline.apply("Read", BigQueryIO.readTableRows()

.fromQuery("SELECT weight_pounds FROM [bigquery-public-data:samples.natality]"))

.apply("MapToDouble", MapElements
.into(TypeDescriptors.doubles())

.via(row -> {
Object obj = row.get("weight_pounds");

return (obj == null ? 0.0 : (Double) obj);
}))
.apply("Top", Top.largest(100))

.apply("MapToString", MapElements
.into(TypeDescriptors.strings())

.via(weight -> weight.toString()))
.apply("Write", TextIO.write().to(""));

pipeline.run().waitUntilFinish();
}
}

```

The "" are basic options like project, job name, temp location, etc. Both programs produce identical outputs.

Running these programs launches a DataFlow job on the Google Cloud with the following results (data from the Google Cloud Platform web interface; screenshots attached).

Python:
```

Read Succeeded 1 hr 40 min 40 sec
MapToFloat Succeeded 2 min 43 sec
Top Succeeded 5 min 25 sec
MapToString
Succeeded 0 sec
Write Succeeded 3 sec
```

Java:
```

Read Succeeded 4 min 45 sec
MapToDouble Succeeded 45 sec
Top Succeeded 52 sec
MapToString Succeeded
0 sec
Write Succeeded 1 sec

```

As you can see, there is an enormous performance hit in Python w.r.t. the reading from BigQuery: 1h40m vs less than 5 minutes.

Furthermore the other standard operations (like Top) are also much slower in Python than in Java.

 

Imported from Jira [BEAM-6064](https://issues.apache.org/jira/browse/BEAM-6064). Original Jira may contain additional context.
Reported by: jankuipers.

No repository files or tests are named. Start by running the supplied Python and Java pipelines against the BigQuery natality query on Dataflow, then compare the stage timings and implementation paths; done means the Python performance discrepancy is explained and addressed with supporting measurements.

Written by the indexing model from the issue text.