apache / apache/beam

Support additional configuration in BigQueryServices.insertAll()

Open
#18,151 0 comments 0 reactions 0 assignees View on GitHub
gcp improvement io java P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

ignoreUnknownValues is requested in https://issues.apache.org/jira/browse/BEAM-1267

There are additional configurations that could be useful.
TableDataInsertAllRequest content = new TableDataInsertAllRequest();
content.setSkipInvalidRows();
content.setTemplateSuffix();
content.setKind();

I think we can improve the BigQueryServices interface by define it as:
void insertAll(TableReference ref, Collection request);
and, provided a static method to prepare requests:

List makeInsertBatches(List rowList, @Nullable List insertIdList);
Then, client can set additional config in the returned list.

Imported from Jira [BEAM-1305](https://issues.apache.org/jira/browse/BEAM-1305). Original Jira may contain additional context.
Reported by: peihe0@gmail.com.

Contributor guide

Open the contributing guide

Research direction

Start at the BigQueryServices.insertAll interface and inspect its current implementations and callers. Trace how TableDataInsertAllRequest batches are built, then determine how the requested options should be passed; done means insertAll supports the additional request configuration without breaking existing callers.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.