apache / apache/beam

Better expose BigQuery quota errors to users

Open
#18,729 0 comments 1 reaction 0 assignees View on GitHub
bug gcp io java P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

Currently users simply see following error.

(9e02ab71e036fcc3): Workflow failed. Causes: (9e02ab71e036fbad): S27:Save to BigQuery/BatchLoads/MultiPartitionsReshuffle/GroupByKey/Read****Save to BigQuery/BatchLoads/MultiPartitionsReshuffle/GroupByKey/GroupByWindow****Save to BigQuery/BatchLoads/MultiPartitionsReshuffle/ExpandIterable****Save to BigQuery/BatchLoads/MultiPartitionsWriteTables/ParMultiDo(WriteTables)****Save to BigQuery/BatchLoads/MultiPartitionsWriteTables/WithKeys/AddKeys/Map****Save to BigQuery/BatchLoads/MultiPartitionsWriteTables/Window.Into()/Window.Assign****Save to BigQuery/BatchLoads/MultiPartitionsWriteTables/GroupByKey/Reify****Save to BigQuery/BatchLoads/MultiPartitionsWriteTables/GroupByKey/Write****Save to BigQuery/BatchLoads/ReifyRenameInput/View.AsIterable/View.CreatePCollectionView/ParDo(ToIsmRecordForGlobalWindow) failed., (f88fcdac78f4b497): A work item was attempted 4 times without success. Each time the worker eventually lost contact with the service. The work item was attempted on: ...

(there might be additional info in worker logs).

We should update the BigQuery connector to raise an exception with a proper error message.

Imported from Jira [BEAM-3376](https://issues.apache.org/jira/browse/BEAM-3376). Original Jira may contain additional context.
Reported by: chamikara.

Contributor guide

Open the contributing guide

Research direction

Start with the BigQuery connector and the worker logs referenced in the report. Trace how the quota failure is currently surfaced, then define the expected user-facing exception from the available Jira context. Done means quota failures produce a proper error message instead of the generic workflow failure.

Written by the indexing model from the issue text.

Assessment

Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.