Kafka brokers are not accessible from outside of the cluster because they advertise the InternalIP only

Affected version

nightly (and possibly older versions)

Current and expected behavior

In the test an initial connection through a port forward is used to connect to 9092, but then the Kafka process advertises InternalIPs (10.x.x.x) which cannot be reached from the outside. The k8s nodes have InternalIPs and ExternalIPs (running on GKE). The operator should pick the ExternalIP over the InternalIP here

Starting port-forwarding of port 9092
Creating test data
Writing test data
%3|1679320241.591|FAIL|rdkafka#producer-1| [thrd:10.10.0.3:31313/1002]: 10.10.0.3:31313/1002: Connect to ipv4#10.10.0.3:31313 failed: Connection timed out (after 131226ms in state CONNECT)
% ERROR: Local: Broker transport failure: 10.10.0.3:31313/1002: Connect to ipv4#10.10.0.3:31313 failed: Connection timed out (after 131226ms in state CONNECT)
%3|1679320372.664|FAIL|rdkafka#producer-1| [thrd:10.10.0.3:31313/1002]: 10.10.0.3:31313/1002: Connect to ipv4#10.10.0.3:31313 failed: Connection timed out (after 130972ms in state CONNECT, 1 identical error(s) suppressed)
% ERROR: Local: Broker transport failure: 10.10.0.3:31313/1002: Connect to ipv4#10.10.0.3:31313 failed: Connection timed out (after 130972ms in state CONNECT, 1 identical error(s) suppressed)
% Delivery failed for message: Local: Message timed out

Possible solution

The listener operator should fix this (?)

Additional context

Environment

Would you like to work on fixing this bug?

None

Start in rust/operator/src/kafka_controller.rs around line 773 and inspect how the listener operator selects node addresses. Reproduce the port-forwarded connection and check the broker’s advertised address. Done means brokers advertise reachable ExternalIPs rather than 10.x.x.x InternalIPs, allowing the external producer test to connect.