Graylog2 / Graylog2/graylog-plugin-integrations

AWS Kinesis/Cloudwatch scheduler errors

Open
#591 0 comments 0 reactions 0 assignees View on GitHub
aws bug
Dominant language
Java
Stars
17
Forks
14
PR merge metrics
No merged PRs in 30d

Description

## Description
when creating an AWS Kinesis/Cloudwatch input, the input may shows the following server errors logs
this may be causing a high volume of memory consumption

## Steps To Reproduce

1. create a AWS Kinesis/Cloudwatch input
1. observe server logs

## Environment

- Graylog Version: 3.3.6
- Elasticsearch Version: 7.9.1
- MongoDB Version: 3.6
- Browser Version: firefox 81

![Screen Shot 2020-10-05 at 11 57 22 AM](https://user-images.githubusercontent.com/22039395/95111149-b4d42e00-0704-11eb-983c-d45d22b74609.png)

```2020-10-05 11:53:10,490 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:10, serverValue:54}] to localhost:27017
2020-10-05 11:53:10,501 INFO : org.graylog2.inputs.InputStateListener - Input [AWS Kinesis/CloudWatch/5f7b4f768fbf2a5a113815d5] is now STARTING
2020-10-05 11:53:10,510 INFO : org.graylog2.inputs.InputStateListener - Input [AWS Kinesis/CloudWatch/5f7b4f768fbf2a5a113815d5] is now RUNNING
2020-10-05 11:53:10,958 INFO : software.amazon.kinesis.leases.dynamodb.DynamoDBLeaseCoordinator - With failover time 10000 ms and epsilon 25 ms, LeaseCoordinator will renew leases every 3308 ms, takeleases every 20050 ms, process maximum of 2147483647 leases and steal 1 lease(s) at a time.
2020-10-05 11:53:10,975 INFO : software.amazon.kinesis.coordinator.Scheduler - Initialization attempt 1
2020-10-05 11:53:10,975 INFO : software.amazon.kinesis.coordinator.Scheduler - Initializing LeaseCoordinator
2020-10-05 11:53:14,390 INFO : software.amazon.kinesis.coordinator.Scheduler - Syncing Kinesis shard info
2020-10-05 11:53:18,300 INFO : software.amazon.kinesis.coordinator.Scheduler - Starting LeaseCoordinator
2020-10-05 11:53:18,449 INFO : software.amazon.kinesis.coordinator.Scheduler - Initialization complete. Starting worker loop.
2020-10-05 11:53:41,496 INFO : software.amazon.kinesis.coordinator.DiagnosticEventLogger - Current thread pool executor state: ExecutorStateEvent(executorName=SchedulerThreadPoolExecutor, currentQueueSize=0, activeThreads=0, coreThreads=0, leasesOwned=0, largestPoolSize=0, maximumPoolSize=2147483647)
2020-10-05 11:54:11,551 INFO : software.amazon.kinesis.coordinator.Scheduler - No activities assigned
2020-10-05 11:54:11,551 INFO : software.amazon.kinesis.coordinator.DiagnosticEventLogger - Current thread pool executor state: ExecutorStateEvent(executorName=SchedulerThreadPoolExecutor, currentQueueSize=0, activeThreads=0, coreThreads=0, leasesOwned=0, largestPoolSize=0, maximumPoolSize=2147483647)
2020-10-05 11:54:11,551 INFO : software.amazon.kinesis.coordinator.Scheduler - Sleeping ...
2020-10-05 11:54:41,611 INFO : software.amazon.kinesis.coordinator.DiagnosticEventLogger - Current thread pool executor state: ExecutorStateEvent(executorName=SchedulerThreadPoolExecutor, currentQueueSize=0, activeThreads=0, coreThreads=0, leasesOwned=0, largestPoolSize=0, maximumPoolSize=2147483647)
2020-10-05 11:55:11,684 INFO : software.amazon.kinesis.coordinator.DiagnosticEventLogger - Current thread pool executor state: ExecutorStateEvent(executorName=SchedulerThreadPoolExecutor, currentQueueSize=0, activeThreads=0, coreThreads=0, leasesOwned=0, largestPoolSize=0, maximumPoolSize=2147483647)
2020-10-05 11:55:12,686 INFO : software.amazon.kinesis.coordinator.Scheduler - No activities assigned
2020-10-05 11:55:12,686 INFO : software.amazon.kinesis.coordinator.Scheduler - Sleeping ...
2020-10-05 11:55:16,704 WARN : software.amazon.kinesis.metrics.CloudWatchMetricsPublisher - Could not publish 14 datums to CloudWatch
java.util.concurrent.TimeoutException: null
at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1771) ~[?:1.8.0_191]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) ~[?:1.8.0_191]
at software.amazon.kinesis.metrics.CloudWatchMetricsPublisher.blockingExecute(CloudWatchMetricsPublisher.java:87) ~[amazon-kinesis-client-2.2.10.jar:?]
at software.amazon.kinesis.metrics.CloudWatchMetricsPublisher.publishMetrics(CloudWatchMetricsPublisher.java:74) [amazon-kinesis-client-2.2.10.jar:?]
at software.amazon.kinesis.metrics.CloudWatchPublisherRunnable.runOnce(CloudWatchPublisherRunnable.java:138) [amazon-kinesis-client-2.2.10.jar:?]
at software.amazon.kinesis.metrics.CloudWatchPublisherRunnable.run(CloudWatchPublisherRunnable.java:84) [amazon-kinesis-client-2.2.10.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
2020-10-05 11:55:41,747 INFO : software.amazon.kinesis.coordinator.DiagnosticEventLogger - Current thread pool executor state: ExecutorStateEvent(executorName=SchedulerThreadPoolExecutor, currentQueueSize=0, activeThreads=0, coreThreads=0, leasesOwned=0, largestPoolSize=0, maximumPoolSize=2147483647)
2020-10-05 11:56:11,809 INFO : software.amazon.kinesis.coordinator.DiagnosticEventLogger - Current thread pool executor state: ExecutorStateEvent(executorName=SchedulerThreadPoolExecutor, currentQueueSize=0, activeThreads=0, coreThreads=0, leasesOwned=0, largestPoolSize=0, maximumPoolSize=2147483647)
2020-10-05 11:56:13,810 INFO : software.amazon.kinesis.coordinator.Scheduler - No activities assigned
2020-10-05 11:56:13,810 INFO : software.amazon.kinesis.coordinator.Scheduler - Sleeping ...
2020-10-05 11:56:28,235 WARN : software.amazon.kinesis.metrics.CloudWatchMetricsPublisher - Could not publish 4 datums to CloudWatch
java.util.concurrent.TimeoutException: null
at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1771) ~[?:1.8.0_191]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) ~[?:1.8.0_191]
at software.amazon.kinesis.metrics.CloudWatchMetricsPublisher.blockingExecute(CloudWatchMetricsPublisher.java:87) ~[amazon-kinesis-client-2.2.10.jar:?]
at software.amazon.kinesis.metrics.CloudWatchMetricsPublisher.publishMetrics(CloudWatchMetricsPublisher.java:74) [amazon-kinesis-client-2.2.10.jar:?]
at software.amazon.kinesis.metrics.CloudWatchPublisherRunnable.runOnce(CloudWatchPublisherRunnable.java:138) [amazon-kinesis-client-2.2.10.jar:?]
at software.amazon.kinesis.metrics.CloudWatchPublisherRunnable.run(CloudWatchPublisherRunnable.java:84) [amazon-kinesis-client-2.2.10.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
2020-10-05 11:56:41,871 INFO : software.amazon.kinesis.coordinator.DiagnosticEventLogger - Current thread pool executor state: ExecutorStateEvent(executorName=SchedulerThreadPoolExecutor, currentQueueSize=0, activeThreads=0, coreThreads=0, leasesOwned=0, largestPoolSize=0, maximumPoolSize=2147483647)
2020-10-05 11:56:53,373 INFO : org.graylog2.inputs.InputStateListener - Input [AWS Kinesis/CloudWatch/5f7b4f768fbf2a5a113815d5] is now STOPPING
2020-10-05 11:56:53,373 INFO : org.graylog.integrations.aws.transports.KinesisConsumer - Waiting up to 20 seconds for Kinesis Consumer shutdown to complete.
2020-10-05 11:56:53,374 INFO : software.amazon.kinesis.coordinator.Scheduler - Worker shutdown requested.
2020-10-05 11:56:53,904 INFO : software.amazon.kinesis.coordinator.Scheduler - All record processors have been shutdown successfully.
2020-10-05 11:56:53,904 INFO : software.amazon.kinesis.coordinator.Scheduler - Starting worker's final shutdown.
2020-10-05 11:56:53,904 INFO : software.amazon.kinesis.metrics.CloudWatchPublisherRunnable - Shutting down CWPublication thread.
2020-10-05 11:56:54,249 INFO : software.amazon.kinesis.metrics.CloudWatchPublisherRunnable - CWPublication thread finished.
2020-10-05 11:56:54,249 INFO : software.amazon.kinesis.coordinator.Scheduler - Worker loop is complete. Exiting from worker.
2020-10-05 11:56:54,807 WARN : software.amazon.kinesis.metrics.CloudWatchPublisherRunnable - Dropping metrics [software.amazon.kinesis.metrics.MetricDatumWithKey@5a342571, software.amazon.kinesis.metrics.MetricDatumWithKey@67c5fd31, software.amazon.kinesis.metrics.MetricDatumWithKey@6f661af1, software.amazon.kinesis.metrics.MetricDatumWithKey@cdbb7a51, software.amazon.kinesis.metrics.MetricDatumWithKey@1f63051] because CloudWatchPublisherRunnable is shutting down.
2020-10-05 11:56:54,807 WARN : software.amazon.kinesis.metrics.CloudWatchPublisherRunnable - Dropping metrics [software.amazon.kinesis.metrics.MetricDatumWithKey@af256011, software.amazon.kinesis.metrics.MetricDatumWithKey@38d8b02e, software.amazon.kinesis.metrics.MetricDatumWithKey@e79e7291] because CloudWatchPublisherRunnable is shutting down.
2020-10-05 11:56:54,807 WARN : software.amazon.kinesis.metrics.CloudWatchPublisherRunnable - Dropping metrics [software.amazon.kinesis.metrics.MetricDatumWithKey@d733be4e, software.amazon.kinesis.metrics.MetricDatumWithKey@d73c1ff1] because CloudWatchPublisherRunnable is shutting down.
2020-10-05 11:56:54,808 INFO : software.amazon.kinesis.leases.dynamodb.DynamoDBLeaseCoordinator - Worker graylog-node-a06f8344bfa040ceb90e6360f1deddcff126a8f5ca6e1d706652f4b2560fd070 has successfully stopped lease-tracking threads
2020-10-05 11:56:54,812 INFO : org.graylog2.inputs.InputStateListener - Input [AWS Kinesis/CloudWatch/5f7b4f768fbf2a5a113815d5] is now STOPPED
2020-10-05 11:56:54,815 INFO : org.graylog2.inputs.InputStateListener - Input [AWS Kinesis/CloudWatch/5f7b4f768fbf2a5a113815d5] is now TERMINATED```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.