elastic / elastic/logstash

Design: Events eviction and Dead Letter Queue

Open
#5,283 12 comments 0 reactions 0 assignees View on GitHub
design DLQ
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
19h 14m
Merged PRs (30d)
63

Description

In the context of upcoming internal queuing persistence, we will introduce pipeline-wide reliability to make sure that queued events will not be removed until acknowledged as fully processed by the final pipeline stage. This will ensure at-least-once delivery semantic in particular in the case of an unsupervised stop or a crash.

In that context, events flowing in the pipeline will require to **always** be acknowledged in the normal runtime. The only situation where events are not acknowledged is upon an unsupervised stop or a crash. This requirement to always acknowledge events introduces the need to evict events upon processing errors or catched/uncatched exceptions.

These should be the possible outcomes of event processing:

```
1. the process is successful, the event/batch is ACK'ed
2. the process is blocked, ACK cannot happens until the process is unblocked
3. the process crashes with an uncatched exception, the event/batch is ACK'ed and evicted (DLQ)
4. the process catches an exception:
4.1. if there's a retry mechanism, the event/batch is retried
4.1.1. the retry is successful, the event/batch is ACK'ed
4.1.2. the retry fails (max retry?), the event/batch is ACK'ed and evicted (DLQ)
4.2. there's no retry mechanism, the event/batch is ACK'ed and evicted (DLQ)
```

We need to introduce a proper/holistic logstash-wide events eviction mechanism.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.