apache / apache/pulsar

Pulsar function replicas auto scaling

Open
#18,584 4 comments 0 reactions 0 assignees View on GitHub
Stale
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Motivation

Pulsar function and connector in Kubernetes runtime should be able to auto scale based on resource usage or the input topic backlog size or some other metrics. The auto scale should be horizontal scaling.

### Solution

_No response_

### Alternatives

Use the existing Kubernetes HPA. But the state of replicas won't be saved because function stop removes the function statefulset. The parallelism is save in bookkeeper not Kubernetes (upon statefulset removal) in the current implementation.

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start by examining the Kubernetes runtime implementation for Pulsar functions and connectors, including how function statefulsets are created, stopped, and how parallelism is stored in BookKeeper. Define the scaling metrics and a way to preserve replica state across stops; done means horizontal autoscaling works without losing the configured parallelism.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kubernetes
Domain
distributed-systems, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.