Graylog2 / Graylog2/graylog-plugin-integrations

AWS Kinesis/CloudWatch error is filling up the filesystem

Open
#458 1 comment 0 reactions 0 assignees View on GitHub
bug triaged
Dominant language
Java
Stars
17
Forks
14
PR merge metrics
No merged PRs in 30d

Description

For the second time in 2 weeks i had the following error in my graylog-server log file:

```
ERROR [PrefetchRecordsPublisher] shardId-000000000000 : Exception thrown while fetching records from Kinesis software.amazon.awssdk.core.exception.SdkClientException: The channel was closed. This may have been done by the client (e.g. because the request was aborted), by the service (e.g. because the request took too long or the client tried to write on a read-only socket), or by an intermediary party (e.g. because the channel was idle for too long).
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:97) ~[?:?]
at software.amazon.awssdk.core.internal.util.ThrowableUtils.asSdkException(ThrowableUtils.java:98) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryExecutor.retryIfNeeded(AsyncRetryableStage.java:125) ~[?:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryExecutor.lambda$execute$0(AsyncRetryableStage.java:107) ~[?:?]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774) ~[?:1.8.0_242]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750) ~[?:1.8.0_242]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0_242]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[?:1.8.0_242]
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage$ResponseHandler.onError(MakeAsyncHttpRequestStage.java:249) ~[?:?]
at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.handleFailure(NettyRequestExecutor.java:263) ~[?:?]
at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.makeRequestListener(NettyRequestExecutor.java:140) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:551) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.util.concurrent.DefaultPromise.access$200(DefaultPromise.java:35) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.util.concurrent.DefaultPromise$1.run(DefaultPromise.java:501) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:510) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:518) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]
Caused by: java.io.IOException: The channel was closed. This may have been done by the client (e.g. because the request was aborted), by the service (e.g. because the request took too long or the client tried to write on a read-only socket), or by an intermediary party (e.g. because the channel was idle for too long).
at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.decorateException(NettyRequestExecutor.java:277) ~[?:?]
at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.handleFailure(NettyRequestExecutor.java:262) ~[?:?]
... 11 more
Caused by: java.nio.channels.ClosedChannelException
at software.amazon.awssdk.thirdparty.io.netty.handler.codec.http2.Http2StreamChannelBootstrap.findCtx(Http2StreamChannelBootstrap.java:156) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.handler.codec.http2.Http2StreamChannelBootstrap.open(Http2StreamChannelBootstrap.java:120) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.handler.codec.http2.Http2StreamChannelBootstrap.open(Http2StreamChannelBootstrap.java:110) ~[?:?]
at software.amazon.awssdk.http.nio.netty.internal.http2.MultiplexedChannelRecord.lambda$acquireClaimedStream$1(MultiplexedChannelRecord.java:89) ~[?:?]
at software.amazon.awssdk.http.nio.netty.internal.utils.NettyUtils.doInEventLoop(NettyUtils.java:139) ~[?:?]
at software.amazon.awssdk.http.nio.netty.internal.http2.MultiplexedChannelRecord.acquireClaimedStream(MultiplexedChannelRecord.java:82) ~[?:?]
at software.amazon.awssdk.http.nio.netty.internal.http2.MultiplexedChannelRecord.acquireStream(MultiplexedChannelRecord.java:75) ~[?:?]
at software.amazon.awssdk.http.nio.netty.internal.http2.Http2MultiplexedChannelPool.acquireStreamOnInitializedConnection(Http2MultiplexedChannelPool.java:221) ~[?:?]
at software.amazon.awssdk.http.nio.netty.internal.http2.Http2MultiplexedChannelPool.acquire(Http2MultiplexedChannelPool.java:115) ~[?:?]
at software.amazon.awssdk.http.nio.netty.internal.utils.BetterFixedChannelPool.acquire0(BetterFixedChannelPool.java:164) ~[?:?]
at software.amazon.awssdk.http.nio.netty.internal.utils.BetterFixedChannelPool.acquire(BetterFixedChannelPool.java:138) ~[?:?]
at software.amazon.awssdk.http.nio.netty.internal.http2.HttpOrHttp2ChannelPool.acquire0(HttpOrHttp2ChannelPool.java:80) ~[?:?]
at software.amazon.awssdk.http.nio.netty.internal.http2.HttpOrHttp2ChannelPool.lambda$acquire$0(HttpOrHttp2ChannelPool.java:69) ~[?:?]
at software.amazon.awssdk.http.nio.netty.internal.utils.NettyUtils.lambda$doInEventLoop$3(NettyUtils.java:143) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:73) ~[?:?]
... 6 more
```

## Expected Behavior
Graylog should handle the exception and not repeat the error indefinitely

## Current Behavior
This error is logged every x msec until the filesystem is full (~ 1GB every 10 minutes)

## Possible Solution

## Steps to Reproduce (for bugs)

## Context

## Your Environment
Ec2 instance inside a vpc using nat gateways and the default kinesis endpoint.

* Graylog Version: 3.2.4
* Elasticsearch Version: 6.8.8
* MongoDB Version: 4.0.14
* Operating System: Centos 7.7
* Browser version: Firefox

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.