How to get failed insert record for file load insertion in BigQuery.
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
I'm using Apache Beam (Java SDK) to insert record in BigQuery using Batch load method (File loads). I want to retrieve those records which failed during insertion.
Is it possible to have a retry policy on failed records?
Actually getting error after retying 1000 times.
Below is my code:
``
```
public static void insertToBigQueryDataLake(
final PCollectionTuple dataStoresCollectionTuple,
final TupleTag> dataLakeValidTag,
final Long loadJobTriggerFrequency,
final Integer loadJobNumShard) {
WriteResult writeResult = dataStoresCollectionTuple
.get(dataLakeValidTag)
.apply(TRANSFORMATION_NAME, DataLakeTableProcessor.dataLakeTableProcessorTransform())
.apply(
WRITING_EVENTS_NAME,
BigQueryIO.>write()
.withMethod(BigQueryIO.Write.Method.FILE_LOADS)
.withTriggeringFrequency(Duration.standardMinutes(loadJobTriggerFrequency))
.withNumFileShards(loadJobNumShard)
.to(new
DynamicTableRowDestinations<>(IS_DATA_LAKE))
.withFormatFunction(BigQueryServiceImpl::dataLakeTableRow));
writeResult.getFailedInserts().apply(ParDo.of(new DoFn() {
@ProcessElement
public void processElement(final ProcessContext processContext) throws IOException {
System.out.println("Table Row : " + processContext.element().toPrettyString());
}
}));
}
```
``
Imported from Jira [BEAM-12633](https://issues.apache.org/jira/browse/BEAM-12633). Original Jira may contain additional context.
Reported by: Ashutoshdixit604.
Contributor guide
Research direction
Start with the Java SDK entry points shown in the example: BigQueryIO.Write.Method.FILE_LOADS and WriteResult.getFailedInserts(). Trace how failed rows and retry exhaustion are handled for file-load insertion, including the dynamic table destination path. Done means the supported way to retrieve failed records and apply a retry policy is established and covered by an appropriate test or documentation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, java
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100