Better expose BigQuery quota errors to users
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
Currently users simply see following error.
(9e02ab71e036fcc3): Workflow failed. Causes: (9e02ab71e036fbad): S27:Save to BigQuery/BatchLoads/MultiPartitionsReshuffle/GroupByKey/Read****Save to BigQuery/BatchLoads/MultiPartitionsReshuffle/GroupByKey/GroupByWindow****Save to BigQuery/BatchLoads/MultiPartitionsReshuffle/ExpandIterable****Save to BigQuery/BatchLoads/MultiPartitionsWriteTables/ParMultiDo(WriteTables)****Save to BigQuery/BatchLoads/MultiPartitionsWriteTables/WithKeys/AddKeys/Map****Save to BigQuery/BatchLoads/MultiPartitionsWriteTables/Window.Into()/Window.Assign****Save to BigQuery/BatchLoads/MultiPartitionsWriteTables/GroupByKey/Reify****Save to BigQuery/BatchLoads/MultiPartitionsWriteTables/GroupByKey/Write****Save to BigQuery/BatchLoads/ReifyRenameInput/View.AsIterable/View.CreatePCollectionView/ParDo(ToIsmRecordForGlobalWindow) failed., (f88fcdac78f4b497): A work item was attempted 4 times without success. Each time the worker eventually lost contact with the service. The work item was attempted on: ...
(there might be additional info in worker logs).
We should update the BigQuery connector to raise an exception with a proper error message.
Imported from Jira [BEAM-3376](https://issues.apache.org/jira/browse/BEAM-3376). Original Jira may contain additional context.
Reported by: chamikara.
Contributor guide
Research direction
Start with the BigQuery connector and the worker logs referenced in the report. Trace how the quota failure is currently surfaced, then define the expected user-facing exception from the available Jira context. Done means quota failures produce a proper error message instead of the generic workflow failure.
Written by the indexing model from the issue text.
Assessment
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100