BigQueryIO - Can't use DynamicDestination with CREATE_IF_NEEDED for unbounded PCollection and FILE_LOADS
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
My workflow : KAFKA -\> Dataflow streaming -\> BigQuery
Given that having low-latency isn't important in my case, I use FILE_LOADS to reduce the costs. I'm using _BigQueryIO.Write_ with a _DynamicDestination_, which is a table with the current hour as a suffix.
This _BigQueryIO.Write_ is configured like this :
```
.withCreateDisposition(CreateDisposition.CREATE_IF_NEEDED)
.withMethod(Method.FILE_LOADS)
.withTriggeringFrequency(triggeringFrequency)
.withNumFileShards(100)
```
The first table is successfully created and is written to. But then the following tables are never created and I get these exceptions:
```
(99e5cd8c66414e7a): java.lang.RuntimeException: Failed to create load job with id prefix 5047f71312a94bf3a42ee5d67feede75_5295fbf25e1a7534f85e25dcaa9f4986_00001_00023,
reached max retries: 3, last failed load job: {
"configuration" : {
"load" : {
"createDisposition"
: "CREATE_NEVER",
"destinationTable" : {
"datasetId" : "dev_mydataset",
"projectId"
: "myproject-id",
"tableId" : "mytable_20180302_16"
},
```
The _CreateDisposition_ used is _CREATE_NEVER_, contrary as _CREATE_IF_NEEDED_ as specified.
Imported from Jira [BEAM-3772](https://issues.apache.org/jira/browse/BEAM-3772). Original Jira may contain additional context.
Reported by: benjben.
Contributor guide
Research direction
The reported entry point is BigQueryIO.Write configured with DynamicDestination, FILE_LOADS, and CREATE_IF_NEEDED. Start by tracing how the disposition is carried into load jobs for later dynamic tables; done means subsequent hourly tables are created as configured rather than submitted with CREATE_NEVER.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, java, kafka
- Domain
- data, databases, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100