apache / apache/beam

The dataflow failure log doesn't include error message

Open
#25,372 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

In a customer case investigation, we got this from the log:
Append to stream projects/optimizely-edp-prod-data-1001/datasets/xxxx/tables/conversions_attributes/streams/_default by client #0 failed with error, operations will be retried. Details: com.google.cloud.bigquery.storage.v1.ConnectionWorker.requestCallback(ConnectionWorker.java:656) com.google.cloud.bigquery.storage.v1.ConnectionWorker.access$100(ConnectionWorker.java:58) com.google.cloud.bigquery.storage.v1.ConnectionWorker$2.run(ConnectionWorker.java:246) com.google.cloud.bigquery.storage.v1.StreamConnection$1.onResponse(StreamConnection.java:63) com.google.cloud.bigquery.storage.v1.StreamConnection$1.onResponse(StreamConnection.java:54) com.google.api.gax.tracing.TracedResponseObserver.onResponse(TracedResponseObserver.java:90) com.google.api.gax.grpc.ExceptionResponseObserver.onResponseImpl(ExceptionResponseObserver.java:74) com.google.api.gax.rpc.StateCheckingResponseObserver.onResponse(StateCheckingResponseObserver.java:60) com.google.api.gax.grpc.GrpcDirectStreamController$ResponseObserverAdapter.onMessage(GrpcDirectStreamController.java:134) io.grpc.ForwardingClientCallListener.onMessage(ForwardingClientCallListener.java:33) io.grpc.ForwardingClientCallListener.onMessage(ForwardingClientCallListener.java:33) io.grpc.ForwardingClientCallListener.onMessage(ForwardingClientCallListener.java:33) io.grpc.internal.DelayedClientCall$DelayedListener.onMessage(DelayedClientCall.java:454) io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInternal(ClientCallImpl.java:662) io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInContext(ClientCallImpl.java:647) io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748),no error

It seems only the stacktrace was provided but not the error message. It makes understanding what's going on very difficult. Could we fix the error logging to include error message?

Contributor guide

Open the contributing guide

Research direction

The report names ConnectionWorker.requestCallback and related Java stack frames, but no Beam source file or test. Start by tracing the Dataflow/BigQuery Storage write failure logging around that callback; done means the emitted failure log contains the error message as well as the stack trace.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, java
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.