fluent / fluent/fluent-plugin-kafka

Add support for MSK IAM authentication in `rdkafka2` output type

Open
#522 3 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Ruby
Stars
307
Forks
178
Avg merge
10h 37m
Merged PRs (30d)
13

Description

### Is your feature request related to a problem? Please describe.

One of the supported authentication & authorization methods for AWS MSK is IAM based authentication with OAUTHBEARER sending in a signed token. This is currently not supported by the plugin so there is no way to authenticate FluentD to send messages to an MSK cluster that only allows that method of authn/z.

Previously, [a similar suggestion was made](https://github.com/fluent/fluent-plugin-kafka/issues/472) but in the context of the `ruby-kafka` library which is deprecated. This feature could, however, be supported using the `rdkafka2` output type.

### Describe the solution you'd like

I created a [fork](https://github.com/madebydna/fluent-plugin-kafka) that adds basic support for MSK IAM authentication with the help of the [aws-msk-iam-sasl-signer-ruby](https://github.com/bruce-szalwinski-he/aws-msk-iam-sasl-signer-ruby) library. I essentially adapted the [example code from the signer library](https://github.com/bruce-szalwinski-he/aws-msk-iam-sasl-signer-ruby/tree/main/examples/rdkafka) for the token generation when using rdkafka as the underlying Kafka library.

However, I'm not sure if this approach is solid enough, so I was wondering if someone could rework the code (or suggest improvements) so that it could be merged into the official plugin. For instance, my solution requires using the shared producer (config `share_producer true`) which is not the default. I wasn't able to use the multi-threaded producer feature as it led to a `ThreadError` due to recursive locking.

### Describe alternatives you've considered

Since I need this feature for use in our production environment, I'm planning on using my own fork for the time being. However, I would be more comfortable if this feature was officially supported and did not require using a modified fork of this plugin.

### Additional context

_No response_

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.