fluent / fluent/fluent-plugin-kafka
Add support for MSK IAM authentication in `rdkafka2` output type
- Dominant language
- Ruby
- Stars
- 307
- Forks
- 178
- Avg merge
- 10h 37m
- Merged PRs (30d)
- 13
Description
### Is your feature request related to a problem? Please describe.
One of the supported authentication & authorization methods for AWS MSK is IAM based authentication with OAUTHBEARER sending in a signed token. This is currently not supported by the plugin so there is no way to authenticate FluentD to send messages to an MSK cluster that only allows that method of authn/z.
Previously, [a similar suggestion was made](https://github.com/fluent/fluent-plugin-kafka/issues/472) but in the context of the `ruby-kafka` library which is deprecated. This feature could, however, be supported using the `rdkafka2` output type.
### Describe the solution you'd like
I created a [fork](https://github.com/madebydna/fluent-plugin-kafka) that adds basic support for MSK IAM authentication with the help of the [aws-msk-iam-sasl-signer-ruby](https://github.com/bruce-szalwinski-he/aws-msk-iam-sasl-signer-ruby) library. I essentially adapted the [example code from the signer library](https://github.com/bruce-szalwinski-he/aws-msk-iam-sasl-signer-ruby/tree/main/examples/rdkafka) for the token generation when using rdkafka as the underlying Kafka library.
However, I'm not sure if this approach is solid enough, so I was wondering if someone could rework the code (or suggest improvements) so that it could be merged into the official plugin. For instance, my solution requires using the shared producer (config `share_producer true`) which is not the default. I wasn't able to use the multi-threaded producer feature as it led to a `ThreadError` due to recursive locking.
### Describe alternatives you've considered
Since I need this feature for use in our production environment, I'm planning on using my own fork for the time being. However, I would be more comfortable if this feature was officially supported and did not require using a modified fork of this plugin.
### Additional context
_No response_
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.