dotnet / dotnet/orleans

Orleans.Streaming.Kafka - Analysis & Plan

Open
#8,494 2 comments 1 reaction 0 assignees View on GitHub
Needs: triage :mag:
Dominant language
C#
Stars
10.9k
Forks
2.1k
Avg merge
13h 56m
Merged PRs (30d)
351

Description

@ReubenBond I initially began the port of [Orleans.Streams.Kafka](https://github.com/jonathansant/Orleans.Streams.Kafka) in one of my branches and started comparing its implementation to the current `Orleans.Streaming.EventHubs` logic - EventHubs and Kafka are [conceptually very similar](https://learn.microsoft.com/en-us/azure/event-hubs/azure-event-hubs-kafka-overview#apache-kafka-and-azure-event-hubs-conceptual-mapping), both are streamed pub-sub solutions using partitioned logs as their backing store. After doing some analysis, I want to hit pause and generate a good gameplan before continuing further.

### Overview
Upon analysis, there are a number of concepts that exist in the EventHubs project that do not yet exist in the Kafka library:

1. The Queue Cache (along with all of its complex logic around pressure monitoring, statistics, purging, etc)
2. Checkpointing
3. Receiving while considering checkpointing and the queue cache

### Proposal

I would like to propose a new library - `Orleans.Streaming.PartitionedLogs` to serve as a common abstraction between these two concepts.

This would allow a great deal of complex (necessarily so) code sharing across Event Hubs and Kafka, rather than being Event Hubs specific - this also paves the way for future integration use cases that are conceptually similar to Event Hubs and Kafka.

I believe we could carefully bring log from EventHubs -> PartitionedLogs without introducing binary breaking changes, etc.

Contributor guide

Open the contributing guide

Research direction

Start by comparing the Orleans.Streams.Kafka implementation with the current Orleans.Streaming.EventHubs logic, focusing on queue caching, checkpointing, and receiving. Evaluate whether a shared Orleans.Streaming.PartitionedLogs library can cover these concepts without binary breaking changes. Done would be an agreed implementation gameplan before further porting work.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, kafka
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.