redpanda-data / redpanda-data/connect
Benthos >4.11.0 does not call ack func when inputs create empty batches
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 8.8k
- Forks
- 969
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 64
Description
It seems that if an input generates an empty batch then the ack func is not called in benthos >4.11.0. Our custom sql_outbox input polls a database table for rows and sets up an ACK func that will ultimately call tx.Commit or tx.Rollback. When there are no results from the database the ack func is never called, the transaction is not closed and connections are leaked until max_conn_open is hit. In Benthos <=4.11.0 the ack func is called even if batch is empty.
This issue started with this commit: https://github.com/benthosdev/benthos/commit/2309559f9a1cf9e49e25af13482fec233cd49f5d
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by examining commit 2309559f9a1cf9e49e25af13482fec233cd49f5d and the input batch acknowledgement path it changed. Reproduce an empty batch from the custom sql_outbox input, then verify that its ACK function is called and that the transaction is closed without leaking connections.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sql
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100