googleapis / googleapis/google-cloud-go

pubsub: No efficient way to get publish errors

Open
#2,953 3 comments 2 reactions 1 assignee Claimed by @hongalex View on GitHub
api: pubsub type: feature request
Dominant language
Go
Stars
4.5k
Forks
1.6k
Avg merge
1d 13h
Merged PRs (30d)
109

Description

**Is your feature request related to a problem? Please describe.**
Currently, `Publish` returns a `PublishResult` which holds a channel. Typically you'll call `Get` to wait for acceptance.

But if you're pumping out large numbers of messages asynchronously, you don't want to wait for acceptance; you just want to "fire and forget". An example is if you want to emit a log entry about an action on every API request; you don't want the logging to slow down the API request. However, you also want to know about any possible errors that occur. So a naive solution is to start a goroutine per publish:

```go
r := topic.Publish(ctx, &pubsub.Message{Data: b})
go func() {
_, err := r.Get(context.Background())
if err != nil {
log.Printf("Could not publish: %s", err)
// ...
}
}()
```

But if you're publishing thousands of messages, you'll have potentially tens or hundreds of thousands of goroutines listening on individual channels, depending on the rate and the latency of Pub/Sub.

You can of course ignore the result altogether. But then you can't know if you're having errors. In particular, there's currently no OpenCensus metric for messages that dropped due to the buffer being full.

It's also possible to add the results to a channel, and then use a smaller number of goroutines to listen to these. But it's hard to do this efficiently without causing queueing and blocking.

**Describe the solution you'd like**
The simplest solution would be to support a per-topic callback for errors:

```go
topic.OnPublishError = func(err error) {
log.Printf("Could not publish: %s", err)
}
```

This means nobody has to wait for anything unless the application explicitly needs to wait for messages to be delivered.

I'd also suggest adding an OpenCensus metric for dropped messages.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.