apache / apache/beam

BigQueryIO - table "Not found" when using BigQueryIO.write() with CREATE_NEVER and WRITE_APPEND[Feature Request]:

Open
#24,001 2 comments 0 reactions 0 assignees View on GitHub
gcp io java new feature P2
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What would you like to happen?

**Problem Encountered:**
For a BigQueryIO.Write configured like in [1], the if target table doesn’t exist, then pipeline throws 404 Table Not Found exception and continuously retries the work item [2].

Whereas for insert errors (broken json or schema error), it is able to catch the error (via getFailedInsertsWithErr)
It was most recently reproduced on Apache Beam SDK for Java 2.39.0

**What you expected to happen:**
Table not found errors should be caught by getFailedInsertsWithErr so that those records can be handled separately (like writing to dead letter queue or to GCS etc.)

[1]
`WriteResult writeResult = results.get(SUCCESS_TAG).apply("WriteSuccessfulRecordsToBQ", BigQueryIO.writeTableRows()
.withMethod(BigQueryIO.Write.Method.STREAMING_INSERTS)
.withFailedInsertRetryPolicy(InsertRetryPolicy.retryTransientErrors()) //Retry all failures except for known persistent errors.
.withWriteDisposition(WRITE_APPEND)
.withCreateDisposition(CREATE_NEVER)
.withExtendedErrorInfo() //- getFailedInsertsWithErr
.ignoreUnknownValues()
.skipInvalidRows()
.withoutValidation()
.to((row) -> {
String tableName = Objects.requireNonNull(row.getValue()).get("event_type").toString();
return new TableDestination(String.format("%s:%s.%s", BQ_PROJECT, BQ_DATASET, tableName), "Some destination");
})`

[2]
`Error message from worker: java.lang.RuntimeException: com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
POST https://bigquery.googleapis.com/bigquery/v2/projects/dfdfdfdfdfd/datasets/sdfsdfdsfsfs/tables/dddddddd/insertAll?prettyPrint=false
{
"code" : 404,
"errors" : [ {
"domain" : "global",
"message" : "Not found: Table dfdfdfdfdfd:sdfsdfdsfsfs.dddddddd",
"reason" : "notFound"
} ],
"message" : "Not found: Table dfdfdfdfdfd:sdfsdfdsfsfs.dddddddd",
"status" : "NOT_FOUND"
}
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:1108)
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:1161)`

### Issue Priority

Priority: 2

### Issue Component

Component: sdk-java-core

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.