AbsaOSS / AbsaOSS/spline

Kafka :: message failure handling

オープン
#1,279 コメント 2 件 リアクション 0 件 担当者 0 名 GitHub で見る
component: Kafka enhancement
主要言語
Scala
スター
667
フォーク
155
平均マージ
3日 6時間
マージ済み PR(30日)
4

説明

# Problem
At the moment the Kafka consumer is implemented in a way that all messages are automatically acknowledged regardless if the processing was successful or not. This would lead to data loss in case of database failure, or a processing timeout due to the message size, for example.

# Goal
We need to implement a robust error handling strategy.

# Solution proposal
Based on the error type, we can decide if the error is persistent or temporary and apply different logic for each type:

### Persistent errors
The error is considered persistent if it's evident or very likely that the error would re-occur on any retry processing attempt in the future. For example, the message is malformed or invalid, or it contradicts the current state of the database (e.g. constraint violation) . In such cases we would acknowledge the message and send it to the dead letters topic, where it would sit and wait manual intervention.

### Temporary errors
It the database is experiencing issues, an unexpected error happens in the app code, or a message dependency aren't fulfilled, we know that the situation that lead to the error is rather temporary and might be fixed soon. In that case we would acknowledge the message and send it to a _retry_ topic, from where the messaged would be automatically pulled and retried periodically.

コントリビューションガイド

このリポジトリのコントリビューションガイドは索引されていません

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。