Graylog2 / Graylog2/graylog2-server

Allow two Graylog clusters to read from same Kinesis steam

Open
#18,149 8 comments 1 reaction 0 assignees View on GitHub
feature triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

## What?
Allow two different Graylog clusters to use the AWS Kinesis CloudWatch Input to read from the same Kinesis stream simultaneously.

## Why?
I am starting a project to migrate from Graylog V3.x to Graylog 5.x clusters. My plan is to run both clusters in parallel so users can use both while they transition all of their Alerts/Dashboards/etc from old to new.

I would like the ability to have an Input on both clusters that read from the same Kinesis stream. As it sits today, it appears that I would have to disable the Input on the old cluster in order for the new cluster to see new messages on the stream as they both use the same DynamoDB table to track where they are in reading data in the stream.

For example, the stream name is eventflow and the DynamoDB table name (for both) is graylog-aws-plugin-eventflow.
Seems like the solution would be to make the DynamoDB table name unique for each Graylog cluster?

Please advise if there is already a way to do this and I have overlooked how to do so.

## Your Environment

* Graylog Version: 5.2.3
* OpenSearch Version: 2.11.1
* MongoDB Version: 6.0.12
* Operating System: Amazon Linux 2
* Browser version: NA

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.