Azure / Azure/azure-functions-host

EventHubs trigger: pass a stable Identifier to EventHubOptions so partitions recover immediately on restart

Open
#11,788 0 comments 0 reactions 0 assignees View on GitHub
Needs: Triage (Functions)
Dominant language
C#
Stars
2k
Forks
482
Avg merge
2d 10h
Merged PRs (30d)
36

Description

## Summary

The Functions host does not set a stable `EventHubOptions.Identifier` that is preserved across restarts of a given host instance. The SDK PR [Azure/azure-sdk-for-net#59416](https://github.com/Azure/azure-sdk-for-net/pull/59416) adds the new `EventHubOptions.Identifier` option that the host can set; the host-side work to assign it a stable value is not yet done.

When multiple workers read from the same Event Hub, they cooperate by writing ownership records to a checkpoint store (a Blob Storage container). A record says, in effect, "worker `X` owns partition `3` as of timestamp `T`," and carries the worker's identifier, today a `Guid` the SDK generates when the processor starts. A partition is only treated as abandoned after its ownership record goes unrefreshed for `PartitionOwnershipExpirationInterval` (default 2 minutes).

[#59416](https://github.com/Azure/azure-sdk-for-net/pull/59416) exposes `EventHubOptions.Identifier` as a pass-through to `EventProcessorOptions.Identifier`. When it is set before the processor starts, the processor recognizes its own prior ownership record on startup (the identifier matches) and resumes immediately. When it is not set, behavior is unchanged: the SDK still auto-generates a `Guid` per instance, exactly like today.

## Motivation

When a Function App node restarts (graceful redeploy, portal restart, node migration, or a crash), the new process starts a fresh processor with a freshly generated `Guid`. From the checkpoint store's perspective, "worker `abc-123` owned partition `3`" and "worker `def-456` just started" are two unrelated workers. The new process has no way to assert that it is the same logical owner as before, so it sees an unexpired record, assumes the prior owner is alive, and waits out the two minutes before claiming the partition. During that window it consumes nothing from those partitions. This matches the customer report [Azure/azure-sdk-for-net#52057](https://github.com/Azure/azure-sdk-for-net/issues/52057).

Once both pieces ship, Event Hubs-triggered Functions recover their partitions immediately on restart instead of waiting roughly two minutes.

## Proposal

The concrete code change lands in the **extension** (`Microsoft.Azure.WebJobs.Extensions.EventHubs`, in `azure-sdk-for-net`), not in this host repo. `EventHubOptions.Identifier` is a bare pass-through with no defaulting, so the work is to pick the stable value and assign it before the processor starts, only when the user has not set one explicitly. The host's contribution is the value itself, `WEBSITE_INSTANCE_ID`, which it already exposes (`EnvironmentSettingNames.AzureWebsiteInstanceId`).

The flow reaches the ownership records: `EventHubOptions.Identifier` writes through to the internal `EventProcessorOptions`, which `EventHubClientFactory` passes as `options:` into `EventProcessorHost`, whose `Identifier` is exactly what gets written as the partition owner in `CheckpointAsync` / `EventProcessorHostPartition.Owner`. Setting it is therefore what a restarted instance compares against.

The extension's existing post-bind hook is the appropriate seam. `AddEventHubs` in `EventHubWebJobsBuilderExtensions.cs` already registers `PostConfigure(ConfigureOptions)`, and that static `ConfigureOptions` method is where derived defaults (offsets, batch sizes) get applied after `host.json` binding. The identifier default folds in there, next to the rest:

```csharp
// In the existing ConfigureOptions(EventHubOptions options) post-configure hook.
if (string.IsNullOrEmpty(options.Identifier))
{
var instanceId = Environment.GetEnvironmentVariable("WEBSITE_INSTANCE_ID");
if (!string.IsNullOrEmpty(instanceId))
{
options.Identifier = instanceId;
}
}
```

Precedence falls out of the ordering. `host.json` binds via `BindOptions()` and the code-level `AddEventHubs(configure)` delegate via `Configure(configure)`, both `IConfigureOptions`; the `ConfigureOptions` hook is `IPostConfigureOptions`, which .NET runs after all of them. An explicit `host.json` value (`extensions.eventHubs.identifier`) wins, then the `configure` delegate, then `WEBSITE_INSTANCE_ID`; otherwise it stays null and the SDK auto-generates a `Guid`, exactly like today. The default is safe and remains overridable.

Two decisions remain open:

- **Where the policy lives.** The snippet above keeps it in the extension, one self-contained change that no-ops off App Service (the env var is empty there). To keep the policy in the host instead, register an `IConfigureOptions` from the Functions host that pulls `AzureWebsiteInstanceId` off its `IEnvironment` abstraction rather than reading the env var inline. Both are valid; the extension-side change is smaller.
- **Uniqueness across processors on one instance.** Ownership is scoped per checkpoint container + consumer group, so a single instance id is fine for the common case. If one instance ever runs multiple processors against the same hub and consumer group, compose the identifier (for example `WEBSITE_INSTANCE_ID` + function name) so they do not collide.

## Tracked work

1. **Choose a stable identifier.** It has to be unique per host instance (so two concurrent instances do not claim the same identity) and stable across restarts of that instance (so a restarted instance reclaims its previous identity). First choice: `WEBSITE_INSTANCE_ID`, a per-VM identifier App Service preserves across restarts of the same VM. Fallback: a value derived from slot + site + function name.
2. **Wire it into the trigger config.** When the host configures the Event Hubs binding, set `EventHubOptions.Identifier` to the chosen value. The decision of whether that is unconditional once the SDK option ships, or gated behind a `host.json` toggle that defaults on, is open.
3. **Verify end-to-end.** On a graceful restart of a Function App with an Event Hubs trigger, confirm the restarted instance picks up its prior partitions within a few seconds rather than the current ~2 minutes.

## Related

- Customer report: [Azure/azure-sdk-for-net#52057](https://github.com/Azure/azure-sdk-for-net/issues/52057)
- SDK PR exposing `Identifier`: [Azure/azure-sdk-for-net#59416](https://github.com/Azure/azure-sdk-for-net/pull/59416)
- Complementary host shutdown-ordering bug: [#11256](https://github.com/Azure/azure-functions-host/issues/11256). Fixing it shrinks the window where Identifier-based recovery is the only path back, but does not eliminate it for crash and forced-restart cases.

Contributor guide

Open the contributing guide

Research direction

Start in the azure-sdk-for-net Event Hubs extension, especially EventHubWebJobsBuilderExtensions.cs and its ConfigureOptions post-bind hook; review SDK PR #59416 to confirm the available option and precedence. Done means selecting and wiring a stable instance identifier without overriding explicit configuration, then verifying partition recovery after a graceful restart.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, csharp
Domain
backend, cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.