apache / apache/beam

[Feature Request]: User can specify the project where BigQueryIO.fromQuery() will be executed

Open
#25,346 0 comments 1 reaction 0 assignees View on GitHub
awaiting triage io java new feature P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What would you like to happen?

By default the query is executed in the same project where the job is running. Often a different project (associated with a particular BigQuery reservation) should be used.

The "bigQueryProject" pipeline option can be used to change the execution project. But it would be better to explicitly specify it on BigQueryIO connector, e.g., BigQueryIO.readTableRows().from("...").usingExecutionProject("..."). This will allow per transform configuration and should prevent accidental execution of query in the wrong project because the "bigQueryProject" option applies to all transforms and used for both reads and writes.

### Issue Priority

Priority: 3 (nice-to-have improvement)

### Issue Components

- [ ] Component: Python SDK
- [X] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [X] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.