redpanda-data / redpanda-data/connect
`/ready` reports connected=true while Kafka brokers are unreachable
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 8.8k
- Forks
- 969
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 64
Description
/ready reports connected=true while Kafka brokers are unreachable
Description
Environment
- Redpanda Connect: 4.103.1
- Previous version tested: 4.98.0
- Kubernetes
- Calico CNI
I am testing the behavior of the Redpanda Connect /ready endpoint when the Kafka brokers become unreachable.
According to the Redpanda Connect documentation:
/pingcan be used as a liveness probe as it always returns a 200.
/readycan be used as a readiness probe as it serves a 200 only when both the input and output are connected, otherwise a 503 is returned.
Documentation: Redpanda Connect Monitoring / Health Checks
Steps to Reproduce
-
Deploy Redpanda Connect 4.103.1 with a Kafka/Redpanda input and output.
-
Configure short TCP connection timeouts:
tcp:
connect_timeout: 2s
tcp_user_timeout: 2s
-
Verify that
/readyinitially reports both input and output as connected. -
Use a network policy to intentionally block traffic between the Redpanda Connect pod and the Kafka brokers.
-
Observe that Redpanda Connect repeatedly reports broker connection failures:
level=warning msg="unable to open connection to broker"
err="dial tcp <broker-address>:<port>: i/o timeout"
path=root.input
level=warning msg="unable to open connection to broker"
err="dial tcp <broker-address>:<port>: i/o timeout"
path=root.output
- Continue querying
/readywhile the brokers remain unreachable.
Actual Result
Despite continuous connection timeout errors, /ready continues to return:
{
"statuses": [
{
"path": "input",
"connected": true
},
{
"path": "output",
"connected": true
}
]
}
Expected Result
Based on the documented behavior of /ready, when the input and/or output cannot establish connections, I would expect /ready to return HTTP 503 and report the affected component as disconnected.
Question
Is this behavior expected in Redpanda Connect 4.103.1?
If not, could this be a bug or regression in the /ready connection-status handling?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the /ready handler and the code that reports input and output connection status, then reproduce the issue with Kafka traffic blocked by a network policy. Done means the endpoint reflects the unreachable broker state with HTTP 503 and affected components reported as disconnected, with coverage for this scenario.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kafka, kubernetes
- Domain
- backend, observability
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100