Separate Authentication/Authorization for metrics route

Who is this for and what problem do they have today?

In organizations which handle sensitive data, exposing the admin api to make use of metrics data does not comply with security requirements.

I wish to make use of the prometheus metrics exposed by RedPanda. However, since this is tied to the admin api, as long as you are authenticated (In our case, mTLS), you are exposed to the entirety of the admin api. And from what I gather by my testing, this includes destructive commands

Using Basic Authentication(admin_api_require_auth) does not hold up to our compliancy standards, which requires mTLS. (Example)

Thusly, a faux-authorization or separate authentication block which specifies certain endpoints, would be a possible solution

What are the success criteria?

The MVP of this feature would be to be able to, in configuration of the admin_api listeners, specify the endpoints a listener will give response to.

A configuration example:

  admin_api:
    - address: 0.0.0.0
      port: 9092
      name: mtls_listener
    - address: 0.0.0.0
      port: 9090
      name: metrics_route
  admin_api_tls:
    - name: mtls_listener
      key_file: mtls_server.key
      cert_file: mtls_server.crt
      truststore_file: mtls_ca.crt
      enabled: true
      require_client_auth: true
    - name: metrics_routes
      key_file: mtls_server.key
      cert_file: mtls_server.crt
      truststore_file: mtls_ca.crt
      enabled: true
      require_client_auth: true
      routes:
        - "/metrics"
        - "/public_metrics"

Apologizes for the probable formatting and or syntax errors above, but I think I get my point accross.

With this, a systems integrator could choose to expose the "metrics_route" listener to a broader audience.

An alternative to using separate listeners would be to take inspiration from the cluster setting of "kafka_mtls_principal_mapping_rules" and map users from principal mappings to specific routes.

A more sturdy alternative to this would be to give each listener the ability to make use of ACLs or the like. Though, this is out of our use-case's scope.

Why is solving this problem impactful?

Organizations with strict compliancy rulesets and/or higher security standards, would be able to more simply integrate RedPanda within their environment without having to rely on external authentication/authorization modules, reducing the systems complexity, and by extension of that, increase the security of the system.

Additional notes

Created this issue as per Alex's request: https://redpandacommunity.slack.com/archives/C01AJDUT88N/p1673452470126379?thread_ts=1673441323.867529&cid=C01AJDUT88N

JIRA Link: CORE-1137

Start by reviewing the existing admin_api and admin_api_tls listener configuration and the /metrics endpoint, then read the linked CORE-1137 context. The issue proposes separate listener route restrictions or principal-based authorization, so the implementation boundary and preferred design need maintainer clarification. Done means metrics can be exposed with the intended authentication while other admin endpoints remain inaccessible.