temporalio / temporalio/temporal

partitioned activity taskQueue based on workflowId, for better caching performance in activity workers

Open
#4,038 1 comment 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Go
Stars
23.2k
Forks
1.9k
Avg merge
2d 8h
Merged PRs (30d)
228

Description

Is your feature request related to a problem? Please describe.
Tasks on an activityTaskqueue are randomly fetched by competing activity workers reading from the same queue.
In case of subsequent activity calls on different activity workers, caching per pod does not work effectively.

For Workflow workers, temporal takes care for their cache performance using sticky exectution. For activity workers, a similar concept does not exist without custom code.

The typical workflows that I usually come up with consist of multiple activities which all operate on one entity (e.g. order workflow with activities operating on 1 ecommerce order, user workflow on 1 user, uber driver workflow on 1 driver entity).

The problem exists in the following scenario

  • there is a set of activities on the same task queue, all operating on the same entity (e.g. Order)
  • when the state of the entity is not fully owned by the workflow history alone (but instead e.g. in some data base, and temporal is only passing identifiers)
  • and when there’s a node(/pod)-specific entity cache in place (which is probably not the case for smaller services), for example Ehcache

Describe the solution you'd like
Partition activity task queues based on workflowId, and assign partitions to specific worker instances (similar to partition assignment within a kafka consumer group).

The number of partitions could be either fixed, or dynamic.
A limitation to have smaller or equal number of workers than queue partitions is not strictly required. If the number of consumer exceeds the number of partitions, it would still improve caching if let’s say only 2 worker instances are competing for tasks on the same partition, as opposed to all worker instances read from all partitions.

Such a taskQueue feature should be only optional, and not the default, as only systems with a cache would benefit from it. Otherwise, random or round-robin dispatching would lead to a better load-balancing across the workers

Describe alternatives you've considered
set custom task queue names in workflow code using activity options, like in the fileprocessing example, which requires to have custom logic in

  1. worker setup (to start the activity worker on aunique taskqueue)
  2. workflow code (to set the acivityOptions in workflowmethod dynamically, instead of when creating the Workflow worker)
  3. activity code (to return a unique taskqueue)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the activity task queue entry point and how activity workers currently compete for tasks, then review worker setup and activity options referenced in the issue. Compare the desired workflowId-based partition assignment with the existing dispatch behavior; done means an optional partitioning design is defined with worker assignment and load-balancing behavior clarified.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.