Graylog2 / Graylog2/graylog-plugin-aws

KinesisConsumer application name not unique across multiple instances of graylog

Open
#97 8 comments 0 reactions 0 assignees View on GitHub
feature requirements-feedback triaged
Dominant language
Java
Stars
93
Forks
37
PR merge metrics
No merged PRs in 30d

Description

There is a problem when attempting to use multiple separate instances of graylog configured to communicate with the same Kinesis stream. The generated application name for the Kinesis client ends up being identical across instances ("graylog-plugin-aws-#stream-name#").

This may work for multiple nodes/workers on the same graylog cluster as its not desired to receive duplicate messages. In my circumstance I have 2 separate graylog instances (customer infra & managed support team infra) and both need to receive a copy of each Kinesis message.

A solution is to build a unique application name based on the graylog clusterId (or node-id if clusterid is null)

As a test I augmented KinesisConsumer.java to accept the clusterId as string in the run() method and augmented the application name with this or use the nodeid if clusterId is null.

```
String instanceId = null;
if (clusterId == null) {
instanceId = this.nodeId.toString();
} else {
instanceId = clusterId.toLowerCase().substring(0,8);
}
final String applicationName = String.format(Locale.ENGLISH, "gap-%s-%s", instanceId, kinesisStreamName);
KinesisClientLibConfiguration config = new KinesisClientLibConfiguration(applicationName, kinesisStreamName,
authProvider, workerId);
```
KinesisTransport.java also requried a change to pass in the clusterId value that is now expected..

Previously the dynamoDB in AWS would not even show another table.. now there is a new table for the 2nd graylog instance.

Perhaps there is a more elegant way to fix this?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.