googleapis / googleapis/google-cloud-go
pubsub: No efficient way to get publish errors
- Dominant language
- Go
- Stars
- 4.5k
- Forks
- 1.6k
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 109
Description
**Is your feature request related to a problem? Please describe.**
Currently, `Publish` returns a `PublishResult` which holds a channel. Typically you'll call `Get` to wait for acceptance.
But if you're pumping out large numbers of messages asynchronously, you don't want to wait for acceptance; you just want to "fire and forget". An example is if you want to emit a log entry about an action on every API request; you don't want the logging to slow down the API request. However, you also want to know about any possible errors that occur. So a naive solution is to start a goroutine per publish:
```go
r := topic.Publish(ctx, &pubsub.Message{Data: b})
go func() {
_, err := r.Get(context.Background())
if err != nil {
log.Printf("Could not publish: %s", err)
// ...
}
}()
```
But if you're publishing thousands of messages, you'll have potentially tens or hundreds of thousands of goroutines listening on individual channels, depending on the rate and the latency of Pub/Sub.
You can of course ignore the result altogether. But then you can't know if you're having errors. In particular, there's currently no OpenCensus metric for messages that dropped due to the buffer being full.
It's also possible to add the results to a channel, and then use a smaller number of goroutines to listen to these. But it's hard to do this efficiently without causing queueing and blocking.
**Describe the solution you'd like**
The simplest solution would be to support a per-topic callback for errors:
```go
topic.OnPublishError = func(err error) {
log.Printf("Could not publish: %s", err)
}
```
This means nobody has to wait for anything unless the application explicitly needs to wait for messages to be delivered.
I'd also suggest adding an OpenCensus metric for dropped messages.
Contributor guide
Assessment
This issue has not been assessed yet.