Azure / Azure/azure-documentdb-changefeedprocessor-dotnet

Stream failure doesn't appear to restart the lease

Open
#131 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
51
Forks
24
PR merge metrics
No merged PRs in 30d

Description

We are having an issue where stream processing failures are throwing an exception, restarting the service, appear to reacquire the lease, but the stream basically goes into a mode where it appears alive but nothing is ever received. It *may* be that if the restart occurs inside a certain time window, it thinks the lease is normal, but it ends up in this weird "zombie" mode where nothing is received. Our solution for now has been to restart the service again, which seems to work, suggesting it may be a timing issue.

Any suggestions for how to deal with this appropriately?

I have listed the logs below:

21:04:42 [Information] Partition "0": renewed lease with result True
21:04:50 [Information] Host '"host-4b8033ae-ae23-409b-9e97-2d2c4963eb77"' 1 partitions, 1 hosts, 0 available leases, target = 1, min = 0, max = 0, mine = 1, will try to take 0 lease(s) for myself'.

-- exception --

21:04:54 [Information] Stopping processor...
21:04:54 [Information] Partition load balancer task stopped.

-- service restarted -- (*notice there is no acquire lease info logs though!*)

21:04:59 [Information] Partition "0": renewed lease with result True
21:05:16 [Information] Partition "0": renewed lease with result True
21:05:33 [Information] Partition "0": renewed lease with result True
21:05:50 [Information] Partition "0": renewed lease with result True
21:06:07 [Information] Partition "0": renewed lease with result True
21:06:24 [Information] Partition "0": renewed lease with result True

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.