knative / knative/func

Kafka functions: KEDA scaling with consumer-lag trigger

Open
#3,934 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
365
Forks
223
Avg merge
2d 3h
Merged PRs (30d)
25

Description

`func deploy` already supports `--deployer keda` for KEDA-based scaling. For Kafka functions, add a Kafka consumer-lag trigger so KEDA can scale based on consumer group lag — enabling scale-to-zero when there are no messages and scaling up under load. Currently Kafka functions require `min-scale: 1` to stay running.

Traffic-based autoscaling (e.g. Knative's KPA or KEDA HTTP scaler) does not work for Kafka functions because there is no inbound HTTP traffic — the function pulls messages from Kafka internally. Scaling must be driven by Kafka consumer group lag instead.

Contributor guide

Open the contributing guide

Research direction

Start at the func deploy entry point for --deployer keda and trace how Kafka functions are represented in the deployment configuration. Compare the existing KEDA path with the Kafka consumer-group settings; done means KEDA can use consumer-group lag to scale Kafka functions to zero and up under load.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kafka, kubernetes
Domain
backend, cli, infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.