dapr / dapr/components-contrib

Feature: Allow setting the Event Hubs client Identifier (aka ClientID, InstanceID) in binding/pubsub declaration

Open
#3,556 4 comments 1 reaction 1 assignee Claimed by @yaron2 View on GitHub
good first issue kind/enhancement
Dominant language
Go
Stars
602
Forks
580
Avg merge
4d 9h
Merged PRs (30d)
6

Description

## In what area(s)?

/area runtime
/area operator

AEH Binding, AEH PubSub

## Describe the feature

### Problem Statement

After a rolling upgrade or restart of an application using an Azure Event Hubs binding (or pub sub), a new client identifier (a uuid/guid) is generated by default for the underlying Azure SDK Client if it is not provided (be that the golang, dotnet, java, whatever version.). This same identifier is used in blob storage metadata for checkpoint management to signify ownership by a particular client in a pool of clients (or a singular client.) This is the mechanism by which partition balancing is performed among a group of competing clients for a given consumer group. The default timeout for a lease is two minutes. A client renews its lease by updating the modified time on the placeholder blob.

Using the example of a single client bound to an event hub with 32 partitions, each time you restart, the client will slowly acquire leases on these partitions as _each lease reaches the two minute timeout_. Until a lease is acquired, _no messages in that partition will be processed. As far as this single client is concerned, it has no idea whether the old guid is an existing competing client or it's own former client/owner id. In my experience, it can take up to 5 minutes to reclaim all partitions with a single client in a dev environment. This is painful :/

example metadata signifying client ownership over partition 3 for event hub "device-in"

image

### Solution

Allowing the client to have a "static" identifier means that when the client restarts, it can rapidly begin start receiving messages as it believes it already owns all the leases because the blob metadata has the same identifier.

In production scenarios, typically you would bind the identifier to the pod name in a k8s stateful set (not replica) to ensure heritage for a given client. This should be easily done via a secretRef with an environment bound secret store (or maybe there's a better way?)

### References

In the Go SDK, this property seems to be aliased as InstanceID and ClientID across the client options.

https://github.com/Azure/azure-sdk-for-go/blob/sdk/messaging/azeventhubs/v1.2.2/sdk/messaging/azeventhubs/consumer_client.go#L156

Microsoft SDK quote:

> It is recommended that you set a stable unique identifier for processor instances, as this allows the processor to recover partition ownership when an application or host instance is restarted. It also aids readability in Azure SDK logs and allows for more easily correlating logs to a specific processor instance

## Release Note

I would suggest repeating the Microsoft SDK information, and suggesting using stateful sets and pod names as the identifier for production. A fixed identifier is very useful for dev, especially if you have multiple partitions.

RELEASE NOTE:

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.