[Bug]: [Python SDK][Dataflow] Weird behavior of `WriteToBigQuery` task with destination table property
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
### What happened?
When passing a complete destination table with ``'PROJECT:DATASET.TABLE'`` to the ``table``, the Dataflow Runner pass the BigQuery ``JobID`` to the BigQuery Job in place of the destination table.
Attached :
[Dataflow pipelines options](https://user-images.githubusercontent.com/128686054/227150441-a079396a-b81d-4988-846a-ce9e014952fc.png)
[Bigquery Load job details](https://user-images.githubusercontent.com/128686054/227151752-5fa56880-efe2-487d-8a36-3c2dec8b7180.png)
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [X] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [X] Component: Google Cloud Dataflow Runner
Contributor guide
Research direction
Start with the Python SDK's WriteToBigQuery task and the Google Cloud Dataflow Runner handling of a complete PROJECT:DATASET.TABLE value. Compare the supplied destination table with the attached BigQuery load job details; done means the BigQuery job receives the destination table rather than the Dataflow JobID.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, python
- Domain
- cloud, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100