Graylog2 / Graylog2/graylog2-server

Drain messages API call

Open
#2,982 3 comments 0 reactions 0 assignees View on GitHub
feature triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

## Expected Behavior

Requesting the ability to drain messages from a graylog server.
## Current Behavior

Currently there are 3 options to effect change on a Graylog server
1. Pause Processing which seems to = stop do anything right now but keep the messages already in the journal
2. Mark as dead to the load balancer which seems to = Make the load balancer think you are offline although this only seems to have any effect on the web UI. We use a load balancer in front of our Graylog Servers and the health check is pinging the API page for ELB status. Despite it being marked as down on the load balancer I still see messages getting appended to the journal. This possibly due to the fact that outbound REST calls for messages (like to AWS) don't get affected by the ELB taking the machine out of the pool and/or messages are still managing to get in from some other path.
3. Graceful shutdown which seems to = stop the graylog process but don't drain the messages off to another host.
## Possible Solution

Implement code to disable all inputs from the machines that has been put into a "drain" state but continue to allow for API calls to be made and for the journal to finish processing any messages in the backlog.
## Context

The primary reason for this request is Auto Scaling in AWS. We are running 3 persistent instances of Graylog however there are times when we need to be able to scale up our ingest servers to handle a surge in message load. We intend to use Auto Scaling groups in AWS for this however as the load drops and the instances are no longer needed we need a graceful way to drain the machine during the termination process (using lifecycle hooks and a Lambda based API call to Graylog). The same Lambda function would first "drain" the node, then make regular API calls to the graylog API to confirm the journal backlog is zero before allowing the lifecycle event to complete ensuring we don't lose any messages when scaling down the cluster. We already have custom metrics in Cloudwatch to monitor the journal backlog across all machines which we would use to trigger a scale up event.
## Your Environment
- Graylog Version: Graylog 2.1.1+01d50e5
- Elasticsearch Version: 2.1.4
- MongoDB Version: 3.2
- Operating System: CentOS 7
- Browser version: Chrome

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.