Azure / Azure/azure-documentdb-changefeedprocessor-dotnet

Lease by Physical Partition Key Range

Open
#148 4 comments 1 reaction 0 assignees View on GitHub
Dominant language
C#
Stars
51
Forks
24
PR merge metrics
No merged PRs in 30d

Description

None of the documentation for change feeds mention physical partitions, but those are the leased units. I don't believe any of my collections have more that 1 physical partition, so none of our feeds are load balanced because one host gets the only available lease.

```
DocDBTrace Information: 0 : Source collection: 'dbs/ReleaseService/colls/Logs', 1 partition(s)
DocDBTrace Information: 0 : Host '1787-4' 1 partitions, 1 hosts, 0 available leases, target = 1, min = 0, max = 0, mine = 1, will try to take 0 lease(s) for myself'.
DocDBTrace Information: 0 : Host '1787-7' 1 partitions, 2 hosts, 0 available leases, target = 1, min = 0, max = 0, mine = 0, will try to take 1 lease(s) for myself'.
DocDBTrace Information: 0 : Host '1787-2' 1 partitions, 2 hosts, 0 available leases, target = 1, min = 0, max = 0, mine = 0, will try to take 1 lease(s) for myself'.
DocDBTrace Information: 0 : Host '1787-9' 1 partitions, 2 hosts, 0 available leases, target = 1, min = 0, max = 0, mine = 0, will try to take 1 lease(s) for myself'.
DocDBTrace Information: 0 : Host '1787-6' 1 partitions, 2 hosts, 0 available leases, target = 1, min = 0, max = 0, mine = 0, will try to take 1 lease(s) for myself'.
DocDBTrace Information: 0 : Host '1787-3' 1 partitions, 2 hosts, 0 available leases, target = 1, min = 0, max = 0, mine = 0, will try to take 1 lease(s) for myself'.
DocDBTrace Information: 0 : Host '1787-1' 1 partitions, 2 hosts, 0 available leases, target = 1, min = 0, max = 0, mine = 0, will try to take 1 lease(s) for myself'.
DocDBTrace Information: 0 : Host '1787-0' 1 partitions, 2 hosts, 0 available leases, target = 1, min = 0, max = 0, mine = 0, will try to take 1 lease(s) for myself'.
DocDBTrace Information: 0 : Host '1787-5' 1 partitions, 2 hosts, 0 available leases, target = 1, min = 0, max = 0, mine = 0, will try to take 1 lease(s) for myself'.
DocDBTrace Information: 0 : Host '1787-8' 1 partitions, 2 hosts, 0 available leases, target = 1, min = 0, max = 0, mine = 0, will try to take 1 lease(s) for myself'.
```

With 2M documents in a collection and throughput at about 150 documents / second, it would take 4 hours for the single active host to process the backlog, and that's with a no-op onChangesDelegate.

Are my collections odd in that they have only one physical?

It would seem like a lot of people assume that change feeds will load balance their partitions given that none of the documentation on change feeds indicates otherwise, but in practice, only one instance processes the entire collection.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.