apache / apache/hertzbeat

[Feature] Support Kafka SCRAM authentication in Kafka Client monitoring

Open
#4,209 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
7.4k
Forks
1.3k
Avg merge
1d 20h
Merged PRs (30d)
32

Description

## Feature Request

Add SCRAM authentication support to the Kafka Client monitoring template and its Java-side AdminClient configuration.

## Is your feature request related to a problem? Please describe

The current Kafka Client template cannot monitor Kafka clusters that require SASL/SCRAM authentication.

Updating `app-kafka_client.yml` alone is insufficient. The YAML template can add authentication fields to the page and pass their values into the kclient protocol configuration, but the Java code that creates Kafka `AdminClient` must also put those values into Kafka `Properties`.

At present, the official implementation only passes the host, port, and `monitorInternalTopic`. The official documentation does not provide SASL parameters, so the built-in template cannot connect to SCRAM-protected Kafka clusters.

Documentation: https://hertzbeat.apache.org/zh-cn/docs/help/kafka_client/

## Describe the solution you would like

Please support SCRAM authentication end to end:

- Add optional authentication fields to `app-kafka_client.yml`, including username, password, SASL mechanism, and security protocol.
- Pass these values through the kclient protocol configuration.
- When creating Kafka `AdminClient`, populate the corresponding Kafka properties, such as `security.protocol`, `sasl.mechanism`, and `sasl.jaas.config`.
- Support at least `SCRAM-SHA-256` and `SCRAM-SHA-512`.
- Keep the existing unauthenticated behavior backward compatible when authentication is not configured.
- Avoid exposing the password in logs or diagnostic output.
- Update the Kafka Client documentation with a SCRAM configuration example.

## Describe alternatives you have considered

Maintaining a locally modified YAML template and Java implementation is possible, but it diverges from the official template and must be maintained across upgrades. YAML-only customization cannot solve the issue because the final Kafka `Properties` are created in Java.

## Additional context

A typical Kafka client configuration requires values equivalent to:

```properties
security.protocol=SASL_PLAINTEXT
sasl.mechanism=SCRAM-SHA-256
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="..." password="...";
```

`SASL_SSL` should also be usable when TLS is required.

Contributor guide

Open the contributing guide

Research direction

Start by reading app-kafka_client.yml and tracing how its values enter the kclient protocol configuration. Then inspect the Java code that creates Kafka AdminClient and the linked Kafka Client documentation. Done means optional SCRAM-SHA-256/512 settings work for SASL_PLAINTEXT and SASL_SSL, unauthenticated behavior remains compatible, passwords are not logged, and the documentation includes an example.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
observability-sre
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.