BrighterCommand / BrighterCommand/Brighter

Make `Partitioner` Required on `KafkaPublication` in v11 and Default Guidance to Murmur2

Open
#4,218 0 comments 0 reactions 0 assignees View on GitHub
.NET 0 - Backlog Breaking Change V11
Dominant language
C#
Stars
2.5k
Forks
296
Avg merge
1d 11h
Merged PRs (30d)
21

Description

### Problem Statement
`KafkaPublication` already exposes a `Partitioner` property, but it **defaults to `ConsistentRandom`** (CRC32-based). Most users never override it, so they silently get librdkafka's CRC32 partitioner without making an explicit choice.

`ConsistentRandom` has weaker distribution properties than `Murmur2` or `Murmur2Random`. Under certain key patterns (sequential IDs, similar prefixes), CRC32 can produce clustering and uneven partition load. Murmur2 provides better statistical spread across partitions.

### Current State
```csharp
public enum Partitioner
{
Random,
Consistent,
ConsistentRandom,
Murmur2,
Murmur2Random,
}

public class KafkaPublication : Publication
{
public Partitioner Partitioner { get; set; } = Partitioner.ConsistentRandom; // silent default
// ...
}
```

### Proposed Change for v11
Remove the default value and make `Partitioner` **required**, forcing the caller to make an explicit choice.

```csharp
public class KafkaPublication : Publication
{
public required Partitioner Partitioner { get; set; }
// ...
}
```

### Recommended Guidance
Update all documentation and samples to recommend **`Partitioner.Murmur2Random`** (or `Murmur2` if null keys are not expected). This provides better distribution across partitions compared to `ConsistentRandom`.

```csharp
new KafkaPublication
{
Topic = new RoutingKey("my.topic"),
Partitioner = Partitioner.Murmur2Random, // explicit, better distribution
// ...
}
```

### Why Murmur2?
- **Better statistical distribution**: Murmur2's avalanche properties spread keys more uniformly than CRC32, reducing hot partitions under common key patterns.
- **Ecosystem alignment**: Most Kafka clients (Java, librdkafka with explicit config) treat murmur2 as the standard for key-based partitioning.

### Migration Path for v11
1. **v11.0.0**:
- Change `Partitioner` from `public Partitioner Partitioner { get; set; } = Partitioner.ConsistentRandom;` to `public required Partitioner Partitioner { get; set; }`.
- Update all Kafka samples to show explicit `Partitioner = Partitioner.Murmur2Random`.
- Update documentation to explain the choice and recommend murmur2.
2. **Migration guide**:
- Every `KafkaPublication` instantiation must add `Partitioner = ...`.
- Recommend `Murmur2Random` for most use cases.

### Benefits
1. **Explicit choice** — no more silent CRC32 default.
2. **Better partition distribution** — murmur2 spreads keys more evenly.
3. **Fewer hot partitions** — improved throughput consistency under load.

Contributor guide

Open the contributing guide

Research direction

Start at the KafkaPublication.Partitioner property and search the repository for every KafkaPublication instantiation, Kafka sample, and related documentation. Make the property required for v11, update each sample and the migration guidance to use an explicit Murmur2 choice, and verify that all documented construction examples provide Partitioner.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.