cockroachdb / cockroachdb/cockroach

cdc: pubsub sink observability

Open
#108,455 2 comments 0 reactions 0 assignees View on GitHub
A-cdc C-enhancement T-cdc
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

pubsub sink observability is lacking.
As observed in #108276, cdc/pubsub/assume-role sometimes fails due to the changefeed
receiving DEADLINE_EXCEEDED error, and thereafter, the changefeed latency measure
recovers somewhat, but not enough for the test to succeed.

The deadline exceeded error likely comes from pubsub client. We don't configure anything
in any special way, but it does appear deadline exceeded errors can be returned:
https://cloud.google.com/pubsub/docs/troubleshooting

```
Publish operations fail with DEADLINE_EXCEEDED
This is likely caused by a client side bottleneck, such as insufficient service CPUs, bad thread health, or network congestion. If a Publish call returns DEADLINE_EXCEEDED, asynchronous Publish calls are being enqueued faster than they are sent to the service, which progressively increases the request latency.
```
Our pubusb sink completely lacks observability into pubsub. No metrics, no logging specific to pubsub sink.
Instead, this error propagates up the stack and the whole job retries. This is suboptimal.

The second part of this request is that we have to look into pubsub client configuration to ensure that it
is configured appropriately. It's possible we may need to either configure (or implement custom) "Retryer" so that
those types of errors are locally retried (i.e. retry each message, w/out transiently failing the entire job).

Jira issue: CRDB-30480

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.