Azure / Azure/azure-functions-java-library

Event Hub Trigger does not process the messages of a partition sequentially

オープン
#152 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
主要言語
Java
スター
45
フォーク
49
PR マージ指標
30日以内にマージされた PR はありません

説明

Hi,

we are using the Event Hub trigger in a Java-based Azure function for processing messages from IoT Hub. We are relying on the fact that message are processed from a partition in the order that they were written to the Event Hub partition, or more specifically that messages with the same partition key are sequentially processed by a single thread. This property of Event Hub partitions is referenced in several places of the documentation, e.g.

> Within each partition, events remain in production order

> Use keys when consumers need to receive events in production order. Since all events with the same key go to the same partition, events with key values can maintain their order during processing"

Source: https://docs.microsoft.com/en-us/azure/architecture/reference-architectures/event-hubs/partitioning-in-event-hubs-and-kafka

> In some scenarios, the ordering of events can be important. For example, you may want your back-end system to process an update command before a delete command. In this scenario, a client application sends events to a specific partition so that the ordering is preserved. When a consumer application consumes these events from the partition, they are read in order

Source: https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-availability-and-consistency?tabs=dotnet

We have found that at least in Java Azure Functions using the Event Hub trigger this is not the case. Instead the messages seem to be dispatched to a thread pool, so that messages of the same partition are often executed concurrently and out of order.

Please advise if this behavior can be changed using configuration. Coming from Apache Kafka that uses the same partitioning and concurrency concept, we were very surprised that upholding the message and processing ordering guarantee per partition is not the default behavior.

コントリビューションガイド

このリポジトリのコントリビューションガイドは索引されていません

調査の方向性

まず、レポートで説明されている Java Azure Functions Event Hub trigger の動作を確認し、パーティション割り当て、thread-pool へのディスパッチ、利用可能な設定に注目してください。同じパーティションのメッセージが同時に、かつ順序どおりでなく実行される可能性があるかを確認し、サポートされている設定、または変更が必要な動作を文書化してください。

索引モデルが issue の本文から書いたものです。

評価

技術スタック
azure, java
領域
backend, cloud
issue の種類
バグ
難易度
4/5
見積もり時間
3〜5日
活発さ
停滞
明瞭さ
説明が足りない
初心者へのやさしさ
25/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。