Simplify Kafka API Listener configuration

Start by reviewing the CORE-907 proposal and the existing Kafka API listener configuration; no source files or tests are identified in the issue. Define the configuration changes needed to combine listener, advertised-listener, TLS, and SASL settings while preserving existing flexibility, then verify the success criteria and update documentation or tests as appropriate.

Who is this for and what problem do they have today?

Everyone who deploys a Redpanda cluster without having an intermediary config abstraction (e.g. K8s operator / custom resource) has to configure the Kafka API listeners. The configuration is very similar to Apache Kafka's listener configuration. However, I think that the configuration could be easier without giving up on the flexibility.

The full listener configuration is spread across multiple blocks and you have to match these blocks by usign a reference

What are the success criteria?

• Configuration is clearer for user (also easier to document)
• Less error prone to configure
• Same flexibility as before so that all usecases can be covered

Why is solving this problem impactful?

Makes Redpanda easier to configure and use / better developer experience at a relatively low implementation cost :-).

Additional notes

Proposed solution:

  kafka_api:
  - name: "kubernetes-internal"
    socket: "0.0.0.0:9092"
    advertised_address: "broker-0.redpanda-headless.svc.cluster.local:9092"
    tls:
      enabled: false
    sasl:
      enabled: false

  - name: "external"
    socket: "0.0.0.0:9093"
    advertised_address: "broker-0.redpanda.com"
    tls:
      enabled: true
      cert_filepath: "/etc/certs/cert.pem"
      key_filepath: "/etc/certs/key.pem"
    sasl:
      enabled: true # should be enabled by default
      mechanism: "SCRAM-SHA-256" # defaults to SCRAM-SHA-256

This merges several blocks that were required to configure a listener config into a single block. Before you had a block for TLS/SASL config, listener config, advertised listener config which were all separate and had to be correlated via config names (which is error prone and not necessarily idiomatic to configure).

The enabled bool on the TLS and SASL block may seem unnecessary, but it in my opinion it fosters clarity by being more explicit here for a single line that has to be written is a trade-of which I'd always go for - but this is opinionated. I like tuning configurations and code for readers and not for writers.

JIRA Link: CORE-907