redpanda-data / redpanda-data/connect

`/ready` reports connected=true while Kafka brokers are unreachable

Open
#4,655 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
8.8k
Forks
969
Avg merge
1d 13h
Merged PRs (30d)
64

Description

/ready reports connected=true while Kafka brokers are unreachable

Description

Environment
  • Redpanda Connect: 4.103.1
  • Previous version tested: 4.98.0
  • Kubernetes
  • Calico CNI

I am testing the behavior of the Redpanda Connect /ready endpoint when the Kafka brokers become unreachable.

According to the Redpanda Connect documentation:

/ping can be used as a liveness probe as it always returns a 200.

/ready can be used as a readiness probe as it serves a 200 only when both the input and output are connected, otherwise a 503 is returned.

Documentation: Redpanda Connect Monitoring / Health Checks

Steps to Reproduce

  1. Deploy Redpanda Connect 4.103.1 with a Kafka/Redpanda input and output.

  2. Configure short TCP connection timeouts:

tcp:
  connect_timeout: 2s
  tcp_user_timeout: 2s
  1. Verify that /ready initially reports both input and output as connected.

  2. Use a network policy to intentionally block traffic between the Redpanda Connect pod and the Kafka brokers.

  3. Observe that Redpanda Connect repeatedly reports broker connection failures:

level=warning msg="unable to open connection to broker"
err="dial tcp <broker-address>:<port>: i/o timeout"
path=root.input

level=warning msg="unable to open connection to broker"
err="dial tcp <broker-address>:<port>: i/o timeout"
path=root.output
  1. Continue querying /ready while the brokers remain unreachable.

Actual Result

Despite continuous connection timeout errors, /ready continues to return:

{
  "statuses": [
    {
      "path": "input",
      "connected": true
    },
    {
      "path": "output",
      "connected": true
    }
  ]
}

Expected Result

Based on the documented behavior of /ready, when the input and/or output cannot establish connections, I would expect /ready to return HTTP 503 and report the affected component as disconnected.

Question

Is this behavior expected in Redpanda Connect 4.103.1?

If not, could this be a bug or regression in the /ready connection-status handling?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the /ready handler and the code that reports input and output connection status, then reproduce the issue with Kafka traffic blocked by a network policy. Done means the endpoint reflects the unreachable broker state with HTTP 503 and affected components reported as disconnected, with coverage for this scenario.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kafka, kubernetes
Domain
backend, observability
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.