BigQueryIO.write() never emits successfully written records
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
In the following code snippet the data gets successfully inserted into BigQuery. But the "getSuccessfulInserts" Transform never returns any data. No log is written.
```
WriteResult writeResultCategory =
categoryWithHeader
.apply("BQ Insert
Category",
BigQueryIO.write()
.withCreateDisposition(CreateDisposition.CREATE_IF_NEEDED)
.withWriteDisposition(WriteDisposition.WRITE_APPEND)
.withMethod(BigQueryIO.Write.Method.STREAMING_INSERTS)
.withExtendedErrorInfo()
.withFailedInsertRetryPolicy(InsertRetryPolicy.neverRetry())
.useBeamSchema()
.to(getTableReference(tableCategory))
);
writeResultCategory
.getSuccessfulInserts()
.apply("Log
successful inserts", ParDo.of(new DoFn() {
@ProcessElement
public void process(ProcessContext c) {
LOG.info("Successfully inserted
Category: " + c.element());
}
}));
```
The problematic code might be in
```
org.apache.beam.sdk.io.gcp.bigquery.BatchedStreamingWrite.BatchAndInsertElements#finishBundle
```
The successfulInserts are never emitted from here.
Please compare this behaviour with the following method where the successfulInserts are emitted:
```
org.apache.beam.sdk.io.gcp.bigquery.BatchedStreamingWrite.InsertBatchedElements#processElement
```
Imported from Jira [BEAM-13464](https://issues.apache.org/jira/browse/BEAM-13464). Original Jira may contain additional context.
Reported by: thokie.
Contributor guide
Assessment
This issue has not been assessed yet.