temporalio / temporalio/temporal

Allow to configure max concurrency of a workflow with the same id

Open
#4,201 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Go
Stars
23.2k
Forks
1.9k
Avg merge
2d 8h
Merged PRs (30d)
228

Description

Is your feature request related to a problem? Please describe.

Hi, I'm curious if someone else run into that use case. We want to ensure that only one workflow with the same ID runs at the same time and that subsequent workflow runs are queued with a concurrency of 1. We are aware that this can be solved with SignalWithStartWorkflow and draining the signal channel but it is very cumbersome to implement. We need to build a queue our-self. We think this mechanism should be provided by temporal natively because it manages the task queue.
Also this approach does not create a new workflow run but extends the history. This is bad for traceability, We want to see a workflow run per job. Is there anything on the roadmap to cover this issue? Therefore are they other approaches to implement this result?

Describe the solution you'd like

Extending client.StartWorkflowOptions with a property e.g. TaskQueueConcurrency to control the desired concurrency.

	workflowOptions := client.StartWorkflowOptions{
		ID:                   "projects/my-branch"
		TaskQueue:            CreateDeploymentTaskQueueName,
		WorkflowIDConcurrencyPolicy: enums.WORKFLOW_CONCURRENCY_POLICY_UNLIMITED | WORKFLOW_CONCURRENCY_POLICY_SERIAL,
	}
	we, err := c.ExecuteWorkflow(context.Background(), workflowOptions, CreateDeploymentWorkflow, options)
  • WORKFLOW_CONCURRENCY_POLICY_UNLIMITED: (default): Runs are scheduled without limits.
  • WORKFLOW_CONCURRENCY_POLICY_SERIAL: Runs with the same ID are queued as separate runs and scheduled when the current workflow run is completed.

Describe alternatives you've considered

Using SignalWithStartWorkflow.

Additional context

https://temporalio.slack.com/archives/CTDTU3J4T/p1682069065643249

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing client.StartWorkflowOptions and the ExecuteWorkflow and SignalWithStartWorkflow entry points mentioned in the issue. Trace how workflow IDs and task queues currently handle concurrent runs, then determine where a serial concurrency policy would need to be represented. Done means separate runs with the same ID are queued and scheduled one at a time, while the default remains unlimited.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.