Graylog2 / Graylog2/graylog2-server
Drain messages API call
- Dominant language
- Java
- Stars
- 8.1k
- Forks
- 1.1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 217
Description
## Expected Behavior
Requesting the ability to drain messages from a graylog server.
## Current Behavior
Currently there are 3 options to effect change on a Graylog server
1. Pause Processing which seems to = stop do anything right now but keep the messages already in the journal
2. Mark as dead to the load balancer which seems to = Make the load balancer think you are offline although this only seems to have any effect on the web UI. We use a load balancer in front of our Graylog Servers and the health check is pinging the API page for ELB status. Despite it being marked as down on the load balancer I still see messages getting appended to the journal. This possibly due to the fact that outbound REST calls for messages (like to AWS) don't get affected by the ELB taking the machine out of the pool and/or messages are still managing to get in from some other path.
3. Graceful shutdown which seems to = stop the graylog process but don't drain the messages off to another host.
## Possible Solution
Implement code to disable all inputs from the machines that has been put into a "drain" state but continue to allow for API calls to be made and for the journal to finish processing any messages in the backlog.
## Context
The primary reason for this request is Auto Scaling in AWS. We are running 3 persistent instances of Graylog however there are times when we need to be able to scale up our ingest servers to handle a surge in message load. We intend to use Auto Scaling groups in AWS for this however as the load drops and the instances are no longer needed we need a graceful way to drain the machine during the termination process (using lifecycle hooks and a Lambda based API call to Graylog). The same Lambda function would first "drain" the node, then make regular API calls to the graylog API to confirm the journal backlog is zero before allowing the lifecycle event to complete ensuring we don't lose any messages when scaling down the cluster. We already have custom metrics in Cloudwatch to monitor the journal backlog across all machines which we would use to trigger a scale up event.
## Your Environment
- Graylog Version: Graylog 2.1.1+01d50e5
- Elasticsearch Version: 2.1.4
- MongoDB Version: 3.2
- Operating System: CentOS 7
- Browser version: Chrome
Contributor guide
Assessment
This issue has not been assessed yet.