dapr / dapr/components-contrib

Support specifying number of partitions for Kafka pubsub component

Open
#3,110 0 comments 0 reactions 0 assignees View on GitHub
help wanted kind/enhancement
Dominant language
Go
Stars
602
Forks
580
Avg merge
4d 9h
Merged PRs (30d)
6

Description

## Describe the feature

[As discussed with Bernd on discord](https://discord.com/channels/778680217417809931/901141467840524289/1146918875443757216)

It would be highly convenient in dev/test scenarios, if dapr provided a metadata configuration to specify the default number of partitions for a given kafka pubsub component (The default would apply to all Topics created within the component)

The kafka server env var to configure the default partition count is `KAFKA_NUM_PARTITIONS` so maybe a familiar metadata property name could be `numPartitions`

```yaml
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
name: kafka-pubsub
spec:
type: pubsub.kafka
version: v1
metadata:
- name: brokers # Required. Kafka broker connection setting
value: "kafka:29092"
- name: consumerGroup
value: "{namespace}"
- name: clientID
value: "workflow-client"
- name: numPartitions
- value: [int, greater than 0] [if not specified, assume default behaviour of sarama client]
```

## Release Note

RELEASE NOTE:

Contributor guide

Open the contributing guide

Research direction

The issue targets the Kafka pubsub component and proposes a numPartitions metadata property, with an example YAML configuration. Start by tracing how Kafka component metadata is parsed and how topics are created, then compare the Sarama client's default behavior. Done means positive values are accepted, omitted metadata preserves existing behavior, and tests cover both cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kafka
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.