googleapis / googleapis/google-cloud-go

bigquery: support extended storage API options for query acceleration

Open
#13,742 6 comments 4 reactions 1 assignee Claimed by @shollyman View on GitHub
api: bigquery priority: p2 type: feature request
Dominant language
Go
Stars
4.5k
Forks
1.6k
Avg merge
1d 13h
Merged PRs (30d)
109

Description

## Is your feature request related to a problem? Please describe.
To get best Arrow read performance from BigQuery Storage Read, we need to be able to enable Arrow compression. We noticed this because the Python client gets nearly twice as much bandwidth as the Go client, and after some benchmarking and trying different things, it turned out that enabling compression was the key difference.

## Describe the solution you'd like
Either always enable compression, or provide an option somewhere to enable compression and choose the type (LZ4 or ZSTD).

## Describe alternatives you've considered
We could fork the client...

## Additional context
In storage_client.go we see `readSession.start` does not support any way of setting `arrow_serialization_options` in `CreateReadSessionRequest`.

https://github.com/googleapis/google-cloud-go/blob/a64705b6d62f4b4e42b870df55d8937b7836b8b7/bigquery/storage_client.go#L138-L165

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.