cockroachdb / cockroachdb/cockroach
cdc: pubsub sink observability
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
pubsub sink observability is lacking.
As observed in #108276, cdc/pubsub/assume-role sometimes fails due to the changefeed
receiving DEADLINE_EXCEEDED error, and thereafter, the changefeed latency measure
recovers somewhat, but not enough for the test to succeed.
The deadline exceeded error likely comes from pubsub client. We don't configure anything
in any special way, but it does appear deadline exceeded errors can be returned:
https://cloud.google.com/pubsub/docs/troubleshooting
```
Publish operations fail with DEADLINE_EXCEEDED
This is likely caused by a client side bottleneck, such as insufficient service CPUs, bad thread health, or network congestion. If a Publish call returns DEADLINE_EXCEEDED, asynchronous Publish calls are being enqueued faster than they are sent to the service, which progressively increases the request latency.
```
Our pubusb sink completely lacks observability into pubsub. No metrics, no logging specific to pubsub sink.
Instead, this error propagates up the stack and the whole job retries. This is suboptimal.
The second part of this request is that we have to look into pubsub client configuration to ensure that it
is configured appropriately. It's possible we may need to either configure (or implement custom) "Retryer" so that
those types of errors are locally retried (i.e. retry each message, w/out transiently failing the entire job).
Jira issue: CRDB-30480
Contributor guide
Assessment
This issue has not been assessed yet.