Change shutdown behaviour when PQ is enabled.
- Dominant language
- Java
- Stars
- 14.9k
- Forks
- 3.5k
- Avg merge
- 19h 14m
- Merged PRs (30d)
- 63
Description
Logstash, by (by default) will not shutdown while events are still in flight. This is to help protect against data loss.
However, when Logstash is used with the Persistent Queue (PQ), the PQ protects against data loss by only acknowledging the batch of events upon close of the batch (which can only happen after reaching the output). Events remain in the PQ until acknowledged.
These two strategies against data loss (shutdown and PQ) are abit at odds with each other. It is not necessary to halt the shutdown process when the PQ is enabled AND events are still in flight since the PQ protects against data loss.
This behavior can have negative consequence in particular when used with centralized configuration management and an output is stalled. For example an ES host moves to a new IP, the output is stalled with inflight events, so you go into the central config management to fix the IP address, however due to the shutdown behavior protecting against data loss, Logstash can not restart the pipeline. This would require an admin to log into every Logstash box and forcibly restart each Logstash instance. This should not necessary when PQ is enabled.
`pipeline.unsafe_shutdown` is nearly identical to what want as the default behavior for the PQ, however, we should expose a configuration to timeout to outputs to finish before forcing the shutdown (to help avoid duplicates). Optionally we could check to see if the inflight event count is going down in addition to the timeout. Also the `pipeline.unsafe_shutdown` name is abit ominous to simply recommend using this option with the PQ.
Note - setting `pipeline.unsafe_shutdown` is set as a startup option when PQ is enabled should be a sufficient workaround, just note that there is a 15 second (hard coded) delay to push healthy data out, else duplicates may happen.
EDIT: Looking at the code abit more... not sure if `pipeline.unsafe_shutdown` is acceptable for pipeline restarts since I don't think the pipeline worker threads that are hung are explicitly killed.
Contributor guide
Assessment
This issue has not been assessed yet.