dapr / dapr/components-contrib
Support specifying number of partitions for Kafka pubsub component
- Dominant language
- Go
- Stars
- 602
- Forks
- 580
- Avg merge
- 4d 9h
- Merged PRs (30d)
- 6
Description
## Describe the feature
[As discussed with Bernd on discord](https://discord.com/channels/778680217417809931/901141467840524289/1146918875443757216)
It would be highly convenient in dev/test scenarios, if dapr provided a metadata configuration to specify the default number of partitions for a given kafka pubsub component (The default would apply to all Topics created within the component)
The kafka server env var to configure the default partition count is `KAFKA_NUM_PARTITIONS` so maybe a familiar metadata property name could be `numPartitions`
```yaml
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
name: kafka-pubsub
spec:
type: pubsub.kafka
version: v1
metadata:
- name: brokers # Required. Kafka broker connection setting
value: "kafka:29092"
- name: consumerGroup
value: "{namespace}"
- name: clientID
value: "workflow-client"
- name: numPartitions
- value: [int, greater than 0] [if not specified, assume default behaviour of sarama client]
```
## Release Note
RELEASE NOTE:
Contributor guide
Research direction
The issue targets the Kafka pubsub component and proposes a numPartitions metadata property, with an example YAML configuration. Start by tracing how Kafka component metadata is parsed and how topics are created, then compare the Sarama client's default behavior. Done means positive values are accepted, omitted metadata preserves existing behavior, and tests cover both cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kafka
- Domain
- backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100