BrighterCommand / BrighterCommand/Brighter
Make `Partitioner` Required on `KafkaPublication` in v11 and Default Guidance to Murmur2
- Dominant language
- C#
- Stars
- 2.5k
- Forks
- 296
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 21
Description
### Problem Statement
`KafkaPublication` already exposes a `Partitioner` property, but it **defaults to `ConsistentRandom`** (CRC32-based). Most users never override it, so they silently get librdkafka's CRC32 partitioner without making an explicit choice.
`ConsistentRandom` has weaker distribution properties than `Murmur2` or `Murmur2Random`. Under certain key patterns (sequential IDs, similar prefixes), CRC32 can produce clustering and uneven partition load. Murmur2 provides better statistical spread across partitions.
### Current State
```csharp
public enum Partitioner
{
Random,
Consistent,
ConsistentRandom,
Murmur2,
Murmur2Random,
}
public class KafkaPublication : Publication
{
public Partitioner Partitioner { get; set; } = Partitioner.ConsistentRandom; // silent default
// ...
}
```
### Proposed Change for v11
Remove the default value and make `Partitioner` **required**, forcing the caller to make an explicit choice.
```csharp
public class KafkaPublication : Publication
{
public required Partitioner Partitioner { get; set; }
// ...
}
```
### Recommended Guidance
Update all documentation and samples to recommend **`Partitioner.Murmur2Random`** (or `Murmur2` if null keys are not expected). This provides better distribution across partitions compared to `ConsistentRandom`.
```csharp
new KafkaPublication
{
Topic = new RoutingKey("my.topic"),
Partitioner = Partitioner.Murmur2Random, // explicit, better distribution
// ...
}
```
### Why Murmur2?
- **Better statistical distribution**: Murmur2's avalanche properties spread keys more uniformly than CRC32, reducing hot partitions under common key patterns.
- **Ecosystem alignment**: Most Kafka clients (Java, librdkafka with explicit config) treat murmur2 as the standard for key-based partitioning.
### Migration Path for v11
1. **v11.0.0**:
- Change `Partitioner` from `public Partitioner Partitioner { get; set; } = Partitioner.ConsistentRandom;` to `public required Partitioner Partitioner { get; set; }`.
- Update all Kafka samples to show explicit `Partitioner = Partitioner.Murmur2Random`.
- Update documentation to explain the choice and recommend murmur2.
2. **Migration guide**:
- Every `KafkaPublication` instantiation must add `Partitioner = ...`.
- Recommend `Murmur2Random` for most use cases.
### Benefits
1. **Explicit choice** — no more silent CRC32 default.
2. **Better partition distribution** — murmur2 spreads keys more evenly.
3. **Fewer hot partitions** — improved throughput consistency under load.
Contributor guide
Research direction
Start at the KafkaPublication.Partitioner property and search the repository for every KafkaPublication instantiation, Kafka sample, and related documentation. Make the property required for v11, update each sample and the migration guidance to use an explicit Murmur2 choice, and verify that all documented construction examples provide Partitioner.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100