apache / apache/beam

BigQueryIO Storage Write API method - suppress logging API errors for known use cases.

Open
#21,219 0 comments 0 reactions 0 assignees View on GitHub
bug gcp io java P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

Many BigQuery Storage API errors get logged at ERROR level even though they are expected and handled by the Beam's transform. They should be suppressed, or output at DEBUG/INFO level to differentiate from abnormal behavior.

Example:

```

{
"jsonPayload":

{ "message": "Got error io.grpc.StatusRuntimeException: ALREADY_EXISTS: The offset
is within stream, expected offset 52125, received 51264 Entity: projects/event-processing-demo/datasets/bigquery_io/tables/events/streams/CiQ2MmZlOTFjNS0wMDAwLTIzNTItOWMxYS01ODI0MjlhOWRiOGM
closing projects/event-processing-demo/datasets/bigquery_io/tables/events/streams/CiQ2MmZlOTFjNS0wMDAwLTIzNTItOWMxYS01ODI0MjlhOWRiOGM",
"step": "Save Rows to BigQuery/StorageApiLoads/StorageApiWriteSharded/Write Records", "worker": "data-processing-streaming-10250846-kps9-harness-hnj3",
"stage": "P6", "job": "2021-10-25_08_46_20-4863073108555028756", "thread": "258", "work": "3800026c09938fac-5cf2f68daacdd",
"logger": "org.apache.beam.sdk.io.gcp.bigquery.StorageApiWritesShardedRecords" }

,
"resource":
{
"type": "dataflow_step",
"labels":

{ "project_id": "event-processing-demo", "step_id": "Save
Rows to BigQuery/StorageApiLoads/StorageApiWriteSharded/Write Records", "job_id": "2021-10-25_08_46_20-4863073108555028756",
"region": "us-central1", "job_name": "data-processing-streaming-storage-write-api-200-2" }

},
"timestamp":
"2021-10-25T15:57:43.617Z",
"severity": "ERROR",
"labels":

{ "compute.googleapis.com/resource_type":
"instance", "dataflow.googleapis.com/job_name": "data-processing-streaming-storage-write-api-200-2",
"dataflow.googleapis.com/log_type": "supportability", "compute.googleapis.com/resource_id": "6365565973148954518",
"dataflow.googleapis.com/job_id": "2021-10-25_08_46_20-4863073108555028756", "dataflow.googleapis.com/region":
"us-central1", "compute.googleapis.com/resource_name": "data-processing-streaming-10250846-kps9-harness-hnj3"
}

,
"logName": "projects/event-processing-demo/logs/dataflow.googleapis.com%2Fworker",
"receiveTimestamp":
"2021-10-25T15:57:47.557344849Z"
}

```

Additional exceptions that fit this category:
* Got error `io.grpc.StatusRuntimeException: FAILED_PRECONDITION: Stream is closed due to com.google.api.gax.rpc.UnavailableException: io.grpc.StatusRuntimeException: UNAVAILABLE: Connection closed after GOAWAY. HTTP/2 error code: NO_ERROR, debug data: server_shutting_down closing`
* Got error `com.google.api.gax.rpc.AbortedException: io.grpc.StatusRuntimeException: ABORTED: Closing the stream because server is restarted`. This is expected and client is advised to reconnect.
* Got error `io.grpc.StatusRuntimeException: FAILED_PRECONDITION: Stream is closed due to com.google.api.gax.rpc.AbortedException: io.grpc.StatusRuntimeException: ABORTED: Closing the stream because server is restarted`. This is expected and client is advised to reconnect.

 

Imported from Jira [BEAM-13162](https://issues.apache.org/jira/browse/BEAM-13162). Original Jira may contain additional context.
Reported by: slilichenko.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.