temporalio / temporalio/temporal

Use different task queue for activity and workflow task retries

Open
#4,600 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Go
Stars
23.2k
Forks
1.9k
Avg merge
2d 8h
Merged PRs (30d)
228

Description

Is your feature request related to a problem? Please describe.
Some applications require different retry behavior based on error codes. For example (source):

Hi, we are currently trying to design retry policy for our activity. The activity is calling http server and we want to retry based on the http status code.

The default for 5xx, it will be retryable with shorter backoff of e.g. 2 sec as these are mostly intermittent.
For 429 (rate limiter) error we want to retry with longer backoff e.g. 1 min.
For 400 error, this might happen because a resource in the server are currently being ‘paused’ by human operator for maintenance. This maintenance is expected to be done < 10 mins, after which the resource is available again. We want to auto retry this request with long backoff e.g. 10 mins.
Other 4xx will NOT be retryable.

Providing different retry options will partially help, but for rate limit error changing backoff of an individual activity is not going to help as the aggregate rate across all of them can still be high.

Describe the solution you'd like
Use a different rate limited (possibly with a dynamic rate limiter) task queue to retry activities in certain scenarios. This way retries of activities that failed with a specific error code would be scheduled in a separate task queue. This allows other activities to execute and retry without being limited.

The queue might not be directly exposed to a user and be an implementation detail.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue does not name files, tests, or entry points. Start by tracing how activity and workflow task retries are scheduled and assigned to task queues, then identify the retry path that would need a separate queue or rate limiter. Done should include the queue behavior described in the issue and coverage for distinct retry handling.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.