conductor-oss / conductor-oss/conductor

Support dynamic per-task rate limiting via StartWorkflowRequest

Open
#790 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
32.2k
Forks
1k
Avg merge
2d 5h
Merged PRs (30d)
41

Description

## Summary

Currently, task rate limits are statically defined in TaskDef. There is no way to override rate limits per workflow execution at runtime. This is needed when different clients or use cases require different rate limits for the same task type.

## Desired Behavior

Allow `StartWorkflowRequest` to include optional `taskRateLimitOverrides` that override static TaskDef rate limits for that workflow execution:

- Overrides keyed by reference task name (most specific) or task definition name
- Task mappers check for overrides before falling back to static TaskDef values
- Existing Redis rate limiting infrastructure works unchanged with dynamic values

## Scope

- `StartWorkflowRequest` — new `taskRateLimitOverrides` field
- `WorkflowModel` — carry overrides through execution
- Task mappers (`SimpleTaskMapper`, `HTTPTaskMapper`, `KafkaPublishTaskMapper`, `UserDefinedTaskMapper`, `DoWhileTaskMapper`) — check overrides
- New `TaskMapperUtils` utility for override resolution
- gRPC proto update for `StartWorkflowRequest`

## Prior Art

Community PR #569 by @hchepey-clari implemented this as a 731-line change with tests. The approach was clean but will need rebasing against current main.

Contributor guide

Open the contributing guide

Research direction

Start with StartWorkflowRequest and the gRPC proto, then trace how WorkflowModel carries execution settings into SimpleTaskMapper, HTTPTaskMapper, KafkaPublishTaskMapper, UserDefinedTaskMapper, and DoWhileTaskMapper. Review the prior PR #569 for implementation context. Done means reference task name overrides take precedence over task definition overrides, all mappers fall back to static TaskDef values, and existing Redis rate limiting remains unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
grpc, java, redis
Domain
api, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.