BrighterCommand / BrighterCommand/Brighter

[Feature] Native Azure Event Hubs messaging gateway (Paramore.Brighter.MessagingGateway.AzureEventHubs)

Open
#4,332 1 comment 0 reactions 1 assignee Claimed by @iancooper View on GitHub
.NET 0 - Backlog feature request V10.X
Dominant language
C#
Stars
2.5k
Forks
296
Avg merge
1d 11h
Merged PRs (30d)
21

Description

**Is your feature request related to a problem? Please describe.**
Brighter has an Azure Service Bus gateway but no Azure Event Hubs gateway (I found no issue, PR or discussion mentioning it). Event Hubs is the standard streaming service on Azure, and MassTransit's Event Hubs rider made "Service Bus for commands, Event Hubs for streams" a common setup. With MassTransit v8 support ending in 2026, this is the first gap MassTransit users on Azure hit when they look at Brighter. In the April 2025 r/dotnet thread you wrote "if there are gaps, transport is easy enough to add. Feel free to raise an issue" — so here it is.

**Describe the solution you'd like**
Treat Azure Event Hubs the way Brighter already treats Kafka: a partitioned, append-only log read by a single-threaded performer that owns partitions, preserves order and checkpoints its own position — not another queue behind the Service Bus gateway.

Event Hubs is Kafka-like, but the mechanics differ: partition load-balancing and checkpointing are client-side in the SDK (`EventProcessorClient` + a Blob Storage checkpoint store), hubs and consumer groups cannot be created from the data plane, and events carry AMQP metadata (sequence number, enqueued time, partition key). There is no dead-letter and no delayed delivery.

That is why MassTransit ships Kafka and Event Hubs as "riders" rather than transports: its bus assumptions (topology, request/response, `_error` queues, scheduling) do not hold on a log, so a rider only has receive endpoints and producers, and after the configured retries it discards the event and moves the checkpoint on. Brighter's performer model seems to fit a log natively, so a gateway is enough.

What are actually the features I would like to use: namespace + `TokenCredential` (EntraID managed identity), a checkpoint container per service, named consumer groups, prefetch and checkpoint interval/count, consumer-side retry and filters, occasionally kill switch and rate limit, raw JSON payloads (non-.NET producers write to the same hubs), producers per hub with partition key and batching, several riders per process. I believe Brighter has counterparts for most of it (performers, pipeline attributes, `UnacceptableMessageLimit`, `JsonMessageMapper`, `PartitionKey`, `CombinedChannelFactory`, DLQ per ADR 0045) except the transport itself and the batch consumer (`Batch`).

**Describe alternatives you've considered**
- The Event Hubs Kafka endpoint with the existing Kafka gateway: needs Standard tier or higher, moves checkpoints from Blob Storage to consumer-group offsets and drops Event Hubs metadata (sequence number, enqueued time).
- Staying on a MassTransit v8.

A native gateway gives one API for Service Bus and Event Hubs, which is what most Azure shops need.
The Event Hubs emulator plus Azurite in docker-compose should make generated gateway tests (ADR 0037) feasible offline.

**Additional context**
Is this something you would consider for V11, as with Azure Storage Queues in #4191, or could an additive gateway land in 10.x? Related roadmap items in #3958: "Universal DLQ for streaming" and the transport generator.
Versions: Brighter 10.7.0, `Azure.Messaging.EventHubs.Processor` 5.12.2.

_Drafted with AI assistance; reviewed by me._

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.