GoogleCloudPlatform / GoogleCloudPlatform/pubsub
Pause/Resume of sink causes publisher to continuously publish
- Dominant language
- Java
- Stars
- 270
- Forks
- 156
- PR merge metrics
- No merged PRs in 30d
Description
Upon initial deployment of sink connector, processing flows as expected. If we pause the connector, wait, and then resume the connector we see an exponential increase in messages sent to PubSub. This is a high(er) volume connector (2.5-4MM messages per day) and when we paused we had about 2.5MM messages ready to be queued in Kafka (it had been paused for roughly 24 hours). Once we resumed the Consumer, PubSub non-Ack'd messages climbed to 9MM from 0 in 4 minutes time.
We believe this has something to do with the messages not being ACK'd from the connector and them being reprocessed immediately. This may be a configuration as we are using the default sink connector config from the repo, as well as the v.5 Alpha uber-jar.
Contributor guide
Assessment
This issue has not been assessed yet.