Graylog2 / Graylog2/graylog-plugin-integrations
AWS Kinesis/CloudWatch error is filling up the filesystem
- Dominant language
- Java
- Stars
- 17
- Forks
- 14
- PR merge metrics
- No merged PRs in 30d
Description
For the second time in 2 weeks i had the following error in my graylog-server log file:
```
ERROR [PrefetchRecordsPublisher] shardId-000000000000 : Exception thrown while fetching records from Kinesis software.amazon.awssdk.core.exception.SdkClientException: The channel was closed. This may have been done by the client (e.g. because the request was aborted), by the service (e.g. because the request took too long or the client tried to write on a read-only socket), or by an intermediary party (e.g. because the channel was idle for too long).
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:97) ~[?:?]
at software.amazon.awssdk.core.internal.util.ThrowableUtils.asSdkException(ThrowableUtils.java:98) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryExecutor.retryIfNeeded(AsyncRetryableStage.java:125) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryExecutor.lambda$execute$0(AsyncRetryableStage.java:107) ~[?:?]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774) ~[?:1.8.0_242]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750) ~[?:1.8.0_242]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0_242]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[?:1.8.0_242]
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage$ResponseHandler.onError(MakeAsyncHttpRequestStage.java:249) ~[?:?]
at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.handleFailure(NettyRequestExecutor.java:263) ~[?:?]
at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.makeRequestListener(NettyRequestExecutor.java:140) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:551) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.util.concurrent.DefaultPromise.access$200(DefaultPromise.java:35) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.util.concurrent.DefaultPromise$1.run(DefaultPromise.java:501) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:510) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:518) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]
Caused by: java.io.IOException: The channel was closed. This may have been done by the client (e.g. because the request was aborted), by the service (e.g. because the request took too long or the client tried to write on a read-only socket), or by an intermediary party (e.g. because the channel was idle for too long).
at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.decorateException(NettyRequestExecutor.java:277) ~[?:?]
at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.handleFailure(NettyRequestExecutor.java:262) ~[?:?]
... 11 more
Caused by: java.nio.channels.ClosedChannelException
at software.amazon.awssdk.thirdparty.io.netty.handler.codec.http2.Http2StreamChannelBootstrap.findCtx(Http2StreamChannelBootstrap.java:156) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.handler.codec.http2.Http2StreamChannelBootstrap.open(Http2StreamChannelBootstrap.java:120) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.handler.codec.http2.Http2StreamChannelBootstrap.open(Http2StreamChannelBootstrap.java:110) ~[?:?]
at software.amazon.awssdk.http.nio.netty.internal.http2.MultiplexedChannelRecord.lambda$acquireClaimedStream$1(MultiplexedChannelRecord.java:89) ~[?:?]
at software.amazon.awssdk.http.nio.netty.internal.utils.NettyUtils.doInEventLoop(NettyUtils.java:139) ~[?:?]
at software.amazon.awssdk.http.nio.netty.internal.http2.MultiplexedChannelRecord.acquireClaimedStream(MultiplexedChannelRecord.java:82) ~[?:?]
at software.amazon.awssdk.http.nio.netty.internal.http2.MultiplexedChannelRecord.acquireStream(MultiplexedChannelRecord.java:75) ~[?:?]
at software.amazon.awssdk.http.nio.netty.internal.http2.Http2MultiplexedChannelPool.acquireStreamOnInitializedConnection(Http2MultiplexedChannelPool.java:221) ~[?:?]
at software.amazon.awssdk.http.nio.netty.internal.http2.Http2MultiplexedChannelPool.acquire(Http2MultiplexedChannelPool.java:115) ~[?:?]
at software.amazon.awssdk.http.nio.netty.internal.utils.BetterFixedChannelPool.acquire0(BetterFixedChannelPool.java:164) ~[?:?]
at software.amazon.awssdk.http.nio.netty.internal.utils.BetterFixedChannelPool.acquire(BetterFixedChannelPool.java:138) ~[?:?]
at software.amazon.awssdk.http.nio.netty.internal.http2.HttpOrHttp2ChannelPool.acquire0(HttpOrHttp2ChannelPool.java:80) ~[?:?]
at software.amazon.awssdk.http.nio.netty.internal.http2.HttpOrHttp2ChannelPool.lambda$acquire$0(HttpOrHttp2ChannelPool.java:69) ~[?:?]
at software.amazon.awssdk.http.nio.netty.internal.utils.NettyUtils.lambda$doInEventLoop$3(NettyUtils.java:143) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:73) ~[?:?]
... 6 more
```
## Expected Behavior
Graylog should handle the exception and not repeat the error indefinitely
## Current Behavior
This error is logged every x msec until the filesystem is full (~ 1GB every 10 minutes)
## Possible Solution
## Steps to Reproduce (for bugs)
## Context
## Your Environment
Ec2 instance inside a vpc using nat gateways and the default kinesis endpoint.
* Graylog Version: 3.2.4
* Elasticsearch Version: 6.8.8
* MongoDB Version: 4.0.14
* Operating System: Centos 7.7
* Browser version: Firefox
Contributor guide
Assessment
This issue has not been assessed yet.