apache / apache/beam

BigQueryIO - Can't use DynamicDestination with CREATE_IF_NEEDED for unbounded PCollection and FILE_LOADS

Open
#18,706 2 comments 0 reactions 0 assignees View on GitHub
bug gcp io java P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

My workflow : KAFKA -\> Dataflow streaming -\> BigQuery

Given that having low-latency isn't important in my case, I use FILE_LOADS to reduce the costs. I'm using _BigQueryIO.Write_ with a _DynamicDestination_, which is a table with the current hour as a suffix.

This _BigQueryIO.Write_ is configured like this :
```

.withCreateDisposition(CreateDisposition.CREATE_IF_NEEDED)
.withMethod(Method.FILE_LOADS)
.withTriggeringFrequency(triggeringFrequency)
.withNumFileShards(100)

```

The first table is successfully created and is written to. But then the following tables are never created and I get these exceptions:

```

(99e5cd8c66414e7a): java.lang.RuntimeException: Failed to create load job with id prefix 5047f71312a94bf3a42ee5d67feede75_5295fbf25e1a7534f85e25dcaa9f4986_00001_00023,
reached max retries: 3, last failed load job: {
"configuration" : {
"load" : {
"createDisposition"
: "CREATE_NEVER",
"destinationTable" : {
"datasetId" : "dev_mydataset",
"projectId"
: "myproject-id",
"tableId" : "mytable_20180302_16"
},

```

The _CreateDisposition_ used is _CREATE_NEVER_, contrary as _CREATE_IF_NEEDED_ as specified.

Imported from Jira [BEAM-3772](https://issues.apache.org/jira/browse/BEAM-3772). Original Jira may contain additional context.
Reported by: benjben.

Contributor guide

Open the contributing guide

Research direction

The reported entry point is BigQueryIO.Write configured with DynamicDestination, FILE_LOADS, and CREATE_IF_NEEDED. Start by tracing how the disposition is carried into load jobs for later dynamic tables; done means subsequent hourly tables are created as configured rather than submitted with CREATE_NEVER.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, java, kafka
Domain
data, databases, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.