apache / apache/beam

Log RPC requests when they fails to improve error messages

Open
#18,037 0 comments 0 reactions 0 assignees View on GitHub
gcp improvement io java P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

For example, BigQuery sometime doesn't provide enough information in the respond. Users can only know what is invalid in the request, if we log the it.

Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 400 Bad Request
{
"code" : 400,
"errors" : [ {
"domain" : "global",
"message" : "Invalid value for: String is not a valid value",
"reason" : "invalid"
} ],
"message" : "Invalid value for: String is not a valid value"
}
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:321)
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1065)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
at com.google.cloud.dataflow.sdk.util.BigQueryTableInserter.tryCreateTable(BigQueryTableInserter.java:427)
at com.google.cloud.dataflow.sdk.util.BigQueryTableInserter.tryCreateTable(BigQueryTableInserter.java:416)
at com.google.cloud.dataflow.sdk.util.BigQueryTableInserter.getOrCreateTable(BigQueryTableInserter.java:373)
at com.google.cloud.dataflow.sdk.io.BigQueryIO$StreamingWriteFn.getOrCreateTable(BigQueryIO.java:2127)
at com.google.cloud.dataflow.sdk.io.BigQueryIO$StreamingWriteFn.finishBundle(BigQueryIO.java:2099)
at com.google.cloud.dataflow.sdk.util.DoFnRunnerBase.finishBundle(DoFnRunnerBase.java:158)

Imported from Jira [BEAM-482](https://issues.apache.org/jira/browse/BEAM-482). Original Jira may contain additional context.
Reported by: peihe0@gmail.com.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the failed request path through BigQueryTableInserter and BigQueryIO, using the Google API exception in the report as the failure example. Determine where the RPC failure is handled and what request details are available there. Done means failed requests provide enough logged information to diagnose invalid values without relying only on the exception message.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, java
Domain
cloud, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.