[QUESTION] Rebalance during transaction breaks exactly-once semantics?
- Ngôn ngữ chính
- Python
- Star
- 1.4k
- Fork
- 269
- Merge trung bình
- 1 ngày 1 giờ
- Pull request đã merge (30 ngày)
- 6
Mô tả
I try to understand how to implement exactly-once guarantees with the [transactional consume-process-produce example](https://aiokafka.readthedocs.io/en/stable/examples/transaction_example.html).
I use the example almost verbatim. I've added a few print statements and added a sleep to simulate a long processing duration.
```python
...
print("Start sleep")
sleep(4)
print("Done sleep. Start commit")
await producer.send_offsets_to_transaction(
commit_offsets, GROUP_ID)
print("Commit done")
...
```
Besides this, I added a `subscribe()` call with my custom `ConsumerRebalanceListener` to see what's happening.
```python
class MyRebalanceListener(ConsumerRebalanceListener):
def on_partitions_assigned(self, partitions):
print("Assigned:", sorted([p.partition for p in partitions]))
def on_partitions_revoked(self, partitions):
print("Revoked:", sorted([p.partition for p in partitions]))
```
My understanding is that the `send_offsets_to_transaction()` should fail if it tries to commit to partitions whose assignment was revoked. To stress test this, I'm adding a worker when the current worker is *sleeping*, i.e after the output was send but before the offsets were committed. I see log output that indicates
```
(Processing of event)
Start sleep
Done sleep. Start commit
Heartbeat failed for group processing-group because it is rebalancing
Revoked: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
Assigned: [0, 2, 4, 6, 8]
Commit done
```
So it seems that the commit succeeded after new partitions were assigned. In the target topic I see duplicated messages, one produced by each worker.
Where does this go wrong? Why doesn't it give me exactly-once guarantees? I can provide the runnable example if this makes it more clear. Shouldn't the transaction fail/be aborted if a rebalance happens in the background?
Hướng dẫn đóng góp
Đánh giá
Issue này chưa được đánh giá.