Graylog2 / Graylog2/graylog2-server

Kinesis/CloudWatch input is noisy when PutMetricData IAM permission missing

Open
#17,648 3 comments 1 reaction 0 assignees View on GitHub
bug triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

The following message was noted in a Cloud tenant's logs:

`2023-12-11 19:45:19,485 WARN : software.amazon.kinesis.metrics.CloudWatchMetricsPublisher - Could not publish 20 datums to CloudWatch
software.amazon.awssdk.services.cloudwatch.model.CloudWatchException: User: arn:aws:iam::redacted:user/graylog is not authorized to perform: cloudwatch:PutMetricData because no identity-based policy allows the cloudwatch:PutMetricData action (Service: CloudWatch, Status Code: 403, Request ID: redacted)
at software.amazon.awssdk.services.cloudwatch.model.CloudWatchException$BuilderImpl.build(CloudWatchException.java:104) ~[graylog.jar:?]
at software.amazon.awssdk.services.cloudwatch.model.CloudWatchException$BuilderImpl.build(CloudWatchException.java:58) ~[graylog.jar:?]
at software.amazon.awssdk.protocols.query.internal.unmarshall.AwsXmlErrorUnmarshaller.unmarshall(AwsXmlErrorUnmarshaller.java:99) ~[graylog.jar:?]
at software.amazon.awssdk.protocols.query.unmarshall.AwsXmlErrorProtocolUnmarshaller.handle(AwsXmlErrorProtocolUnmarshaller.java:102) ~[graylog.jar:?]
at software.amazon.awssdk.protocols.query.unmarshall.AwsXmlErrorProtocolUnmarshaller.handle(AwsXmlErrorProtocolUnmarshaller.java:82) ~[graylog.jar:?]
at software.amazon.awssdk.core.http.MetricCollectingHttpResponseHandler.lambda$handle$0(MetricCollectingHttpResponseHandler.java:52) ~[graylog.jar:?]
at software.amazon.awssdk.core.internal.util.MetricUtils.measureDurationUnsafe(MetricUtils.java:67) ~[graylog.jar:?]
at software.amazon.awssdk.core.http.MetricCollectingHttpResponseHandler.handle(MetricCollectingHttpResponseHandler.java:52) ~[graylog.jar:?]
at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler.lambda$prepare$0(AsyncResponseHandler.java:89) ~[graylog.jar:?]
at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147) ~[?:?]
at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler$BaosSubscriber.onComplete(AsyncResponseHandler.java:132) ~[graylog.jar:?]
at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler$DataCountingPublisher$1.onComplete(ResponseHandler.java:515) ~[graylog.jar:?]
at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.runAndLogError(ResponseHandler.java:250) ~[graylog.jar:?]
at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.access$600(ResponseHandler.java:75) ~[graylog.jar:?]
at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler$PublisherAdapter$1.onComplete(ResponseHandler.java:371) ~[graylog.jar:?]
at software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerPublisher.publishMessage(HandlerPublisher.java:402) ~[graylog.jar:?]
at software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerPublisher.flushBuffer(HandlerPublisher.java:338) ~[graylog.jar:?]
at software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerPublisher.receivedDemand(HandlerPublisher.java:291) ~[graylog.jar:?]
at software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerPublisher.access$200(HandlerPublisher.java:61) ~[graylog.jar:?]
at software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerPublisher$ChannelSubscription$1.run(HandlerPublisher.java:495) ~[graylog.jar:?]
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[graylog.jar:?]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[graylog.jar:?]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[graylog.jar:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566) ~[graylog.jar:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[graylog.jar:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[graylog.jar:?]
at java.lang.Thread.run(Thread.java:833) [?:?]`

Per https://github.com/Graylog2/graylog-plugin-integrations/issues/293#issuecomment-540770927 this is not a blocker or likely to be impacting message pull, and indeed the input seems to be functioning normally.

However, users may not want to give IAM permissions for `cloudwatch:PutMetricData` for whatever reason, so writing metrics back into CloudWatch should be a toggle on the input. This would impact both Cloud and on-prem installations, where this can be quite noisy, as in the Cloud tenant where it was noted, it's creating about 24 log entries per minute, every minute:

![image](https://github.com/Graylog2/graylog2-server/assets/2953243/1c7f5a9a-deff-43fc-8634-6792d0ea1300)

Another idea would be to check for existence of the permission on input start, then do not publish metrics if permission is not available.

## Your Environment

* Graylog Version: Graylog Cloud 5.2.2 (388)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.