Graylog2 / Graylog2/graylog-plugin-integrations

Exception trying to fetch the data from the second Kinesis stream located in another AWS region

Open
#435 10 comments 0 reactions 1 assignee Claimed by @waab76 View on GitHub
aws bug triaged
Dominant language
Java
Stars
17
Forks
14
PR merge metrics
No merged PRs in 30d

Description

Hello.

## Description
The first input was configured to fetch logs from the Kinesis stream in the `us-east-1` AWS region and working now. Then I was able to configure the second input and even received a message to pass the configuration check. But after that no more messages have been received via this input. Second input configuration is the same as the first one except it's located in another `eu-west-2` region .

## Steps To Reproduce

1. In the region `us-east-1`:
* Create EC2 Autoscaling Group (min=1, max=1).
* Create ECS cluster
* Attach EC2 instance to the cluster.
* Create Graylog ECS Task and configure task role to fetch logs from any Kinesis stream.
* Start a ECS Service to run Graylog container on the single EC2 node.

2. Still in the region `us-east-1`:
* Create CloudWatch Log group.
* Create Kinesis stream (1 shard, 24h retention).
* Create CloudWatch subscription.

3. In the region `eu-west-2`:
* Create CloudWatch Log group.
* Create Kinesis stream (1 shard, 24h retention).
* Create CloudWatch subscription.

4. Login into Graylog and configure 2 `AWS Kinesis/CloudWatch` inputs from both aforementioned Kinesis streams.

5. Only the first input will be working. The second throws exception:
```java
2020-03-18 09:03:37,421 ERROR: software.amazon.kinesis.lifecycle.InitializeTask - Caught exception:
software.amazon.awssdk.services.kinesis.model.InvalidArgumentException: StartingSequenceNumber 49603881713016572917281669991559719292395929931979161602 used in GetShardIterator on shard shardId-000000000000 in stream -stg-logs-stream under account is invalid because it did not come from this stream. (Service: Kinesis, Status Code: 400, Request ID: )
at software.amazon.awssdk.services.kinesis.model.InvalidArgumentException$BuilderImpl.build(InvalidArgumentException.java:118) ~[?:?]
at software.amazon.awssdk.services.kinesis.model.InvalidArgumentException$BuilderImpl.build(InvalidArgumentException.java:78) ~[?:?]
at software.amazon.awssdk.protocols.json.internal.unmarshall.AwsJsonProtocolErrorUnmarshaller.unmarshall(AwsJsonProtocolErrorUnmarshaller.java:88) ~[?:?]
at software.amazon.awssdk.protocols.json.internal.unmarshall.AwsJsonProtocolErrorUnmarshaller.handle(AwsJsonProtocolErrorUnmarshaller.java:63) ~[?:?]
at software.amazon.awssdk.protocols.json.internal.unmarshall.AwsJsonProtocolErrorUnmarshaller.handle(AwsJsonProtocolErrorUnmarshaller.java:42) ~[?:?]
at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler.lambda$prepare$0(AsyncResponseHandler.java:88) ~[?:?]
at java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:966) ~[?:1.8.0_242]
at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:940) ~[?:1.8.0_242]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0_242]
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975) ~[?:1.8.0_242]
at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler$BaosSubscriber.onComplete(AsyncResponseHandler.java:129) ~[?:?]
at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.runAndLogError(ResponseHandler.java:171) ~[?:?]
at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.access$500(ResponseHandler.java:68) ~[?:?]
at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler$PublisherAdapter$1.onComplete(ResponseHandler.java:287) ~[?:?]
at software.amazon.awssdk.thirdparty.com.typesafe.netty.HandlerPublisher.complete(HandlerPublisher.java:408) ~[?:?]
at software.amazon.awssdk.thirdparty.com.typesafe.netty.HandlerPublisher.handlerRemoved(HandlerPublisher.java:395) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.channel.AbstractChannelHandlerContext.callHandlerRemoved(AbstractChannelHandlerContext.java:972) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.channel.DefaultChannelPipeline.callHandlerRemoved0(DefaultChannelPipeline.java:641) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.channel.DefaultChannelPipeline.remove(DefaultChannelPipeline.java:481) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.channel.DefaultChannelPipeline.remove(DefaultChannelPipeline.java:427) ~[?:?]
at software.amazon.awssdk.thirdparty.com.typesafe.netty.http.HttpStreamsHandler.removeHandlerIfActive(HttpStreamsHandler.java:328) ~[?:?]
at software.amazon.awssdk.thirdparty.com.typesafe.netty.http.HttpStreamsHandler.handleReadHttpContent(HttpStreamsHandler.java:189) ~[?:?]
at software.amazon.awssdk.thirdparty.com.typesafe.netty.http.HttpStreamsHandler.channelRead(HttpStreamsHandler.java:165) ~[?:?]
at software.amazon.awssdk.thirdparty.com.typesafe.netty.http.HttpStreamsClientHandler.channelRead(HttpStreamsClientHandler.java:148) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[?:?]
at software.amazon.awssdk.http.nio.netty.internal.LastHttpContentHandler.channelRead(LastHttpContentHandler.java:43) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[?:?]
at software.amazon.awssdk.http.nio.netty.internal.http2.Http2ToHttpInboundAdapter.onDataRead(Http2ToHttpInboundAdapter.java:66) ~[?:?]
at software.amazon.awssdk.http.nio.netty.internal.http2.Http2ToHttpInboundAdapter.channelRead0(Http2ToHttpInboundAdapter.java:44) ~[?:?]
at software.amazon.awssdk.http.nio.netty.internal.http2.Http2ToHttpInboundAdapter.channelRead0(Http2ToHttpInboundAdapter.java:38) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.handler.codec.http2.AbstractHttp2StreamChannel$Http2ChannelUnsafe.doRead0(AbstractHttp2StreamChannel.java:851) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.handler.codec.http2.AbstractHttp2StreamChannel$Http2ChannelUnsafe.doBeginRead(AbstractHttp2StreamChannel.java:793) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.handler.codec.http2.AbstractHttp2StreamChannel$Http2ChannelUnsafe.beginRead(AbstractHttp2StreamChannel.java:765) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.read(DefaultChannelPipeline.java:1374) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeRead(AbstractChannelHandlerContext.java:685) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.channel.AbstractChannelHandlerContext.read(AbstractChannelHandlerContext.java:670) ~[?:?]
at software.amazon.awssdk.thirdparty.com.typesafe.netty.HandlerPublisher.requestDemand(HandlerPublisher.java:86) ~[?:?]
at software.amazon.awssdk.thirdparty.com.typesafe.netty.http.HttpStreamsHandler$1.requestDemand(HttpStreamsHandler.java:157) ~[?:?]
at software.amazon.awssdk.thirdparty.com.typesafe.netty.HandlerPublisher.flushBuffer(HandlerPublisher.java:311) ~[?:?]
at software.amazon.awssdk.thirdparty.com.typesafe.netty.HandlerPublisher.receivedDemand(HandlerPublisher.java:258) ~[?:?]
at software.amazon.awssdk.thirdparty.com.typesafe.netty.HandlerPublisher.access$200(HandlerPublisher.java:41) ~[?:?]
at software.amazon.awssdk.thirdparty.com.typesafe.netty.HandlerPublisher$ChannelSubscription$1.run(HandlerPublisher.java:452) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:510) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:518) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044) ~[?:?]
at software.amazon.awssdk.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]
```

## Environment

- Graylog Version: `3.2.2+2f9a628, codename Ethereal Elk`
- JVM: `PID 19, Oracle Corporation 1.8.0_242 on Linux 4.14.165-133.209.amzn2.x86_64`
- Installed plugins:
* AWS plugins: `3.2.2`
* Collector: `3.2.2`
* Integrations: `3.2.2`
* Threat Intelligence Plugin: `3.2.2`

- Elasticsearch Version: `[LTm5Sek] version[6.8.5], pid[1], build[oss/docker/78990e9/2019-11-13T20:04:24.100411Z], OS[Linux/4.14.165-133.209.amzn2.x86_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/13.0.1/13.0.1+9]`
- MongoDB Version: `3.6.17`
- Browser Version: `Google Chrome | 79.0.3945.88 (Official Build) (64-bit)`

Thank you!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.