googleapis / googleapis/google-cloud-go
pubsub: `(*Subscription).BatchReceive`
- Dominant language
- Go
- Stars
- 4.5k
- Forks
- 1.6k
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 109
Description
**Is your feature request related to a problem? Please describe.**
Often when receiving messages on pubsub the application need to perform some operation on a remote service (e.g. write to a database, or call a different service). Often this kind of operations can be more efficiently performed in batch, so instead of performing N inserts/updates of 1 row in the database we perform 1 insert/update of N rows. Currently `(*Subscription).Receive` is not really amenable to this usage pattern, as each message is normally passed to separate invocations of the handler, each running in its own goroutine.
**Describe the solution you'd like**
A `BatchReceive` API that would allow to specify batch limits (number of messages, number of bytes, max wait for additional messages, max number of concurrent handler invocations) and then would pass an appropriately-sized batch to the handler. The handler should then Ack/Nack each individual message as appropriate.
Example usage would be something like
```
batchCfg := BatchConfig{
// start a batch when it reaches 1000 messages, or when 100ms have
// elapsed since the first message was received; do not start more than
// 2 batches concurrently
MaxMessages: 1000,
MaxWait: 100*time.Millisecond,
MaxConcurrentHandlers: 2,
}
// alternatively BatchConfig may also be part of the Subscription configuration
err := sub.BatchReceive(ctx, batchCfg, func(ctx context.Context, msgs []*Message) {
// Process each *Message in msgs. Ack/Nack each as needed.
// len(msgs) is guaranteed to be between 1 and MaxMessages, inclusive.
// This handler is guaranteed to not have more than MaxConcurrentHandlers concurrent invocations.
})
```
**Describe alternatives you've considered**
Implementing this as a utility function outside of this library is possible, but it's prone to misuse (e.g. if the limits on the subscription are smaller than the limits specified in the batching configuration) and quite wasteful (as the pubsub library will still create N goroutines, that will all have to block and wait for the batch to complete anyway).
**Additional context**
Contributor guide
Assessment
This issue has not been assessed yet.