apache / apache/beam

How to get failed insert record for file load insertion in BigQuery.

Open
#21,086 0 comments 0 reactions 0 assignees View on GitHub
gcp improvement io java P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

I'm using Apache Beam (Java SDK) to insert record in BigQuery using Batch load method (File loads). I want to retrieve those records which failed during insertion.

Is it possible to have a retry policy on failed records?

Actually getting error after retying 1000 times.

Below is my code:

 

``
```

public static void insertToBigQueryDataLake(
final PCollectionTuple dataStoresCollectionTuple,

final TupleTag> dataLakeValidTag,
final Long loadJobTriggerFrequency,

final Integer loadJobNumShard) {

WriteResult writeResult = dataStoresCollectionTuple

.get(dataLakeValidTag)
.apply(TRANSFORMATION_NAME, DataLakeTableProcessor.dataLakeTableProcessorTransform())

.apply(
WRITING_EVENTS_NAME,
BigQueryIO.>write()
.withMethod(BigQueryIO.Write.Method.FILE_LOADS)

.withTriggeringFrequency(Duration.standardMinutes(loadJobTriggerFrequency))

.withNumFileShards(loadJobNumShard)
.to(new
DynamicTableRowDestinations<>(IS_DATA_LAKE))
.withFormatFunction(BigQueryServiceImpl::dataLakeTableRow));

writeResult.getFailedInserts().apply(ParDo.of(new DoFn() {
@ProcessElement

public void processElement(final ProcessContext processContext) throws IOException {

System.out.println("Table Row : " + processContext.element().toPrettyString());
}

}));

}
```

``

Imported from Jira [BEAM-12633](https://issues.apache.org/jira/browse/BEAM-12633). Original Jira may contain additional context.
Reported by: Ashutoshdixit604.

Contributor guide

Open the contributing guide

Research direction

Start with the Java SDK entry points shown in the example: BigQueryIO.Write.Method.FILE_LOADS and WriteResult.getFailedInserts(). Trace how failed rows and retry exhaustion are handled for file-load insertion, including the dynamic table destination path. Done means the supported way to retrieve failed records and apply a retry policy is established and covered by an appropriate test or documentation.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, java
Domain
data-engineering, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.