temporalio / temporalio/temporal

Matching service optimization - Do not persist sticky workflow task

Open
#3,168 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Go
Stars
23.2k
Forks
1.9k
Avg merge
2d 8h
Merged PRs (30d)
228

Description

Sticky workflow tasks usually have short timeout, in terms of seconds.

When a sticky workflow task cannot be sync-matched, matching service right now will persist the task into DB and wait for SDK with local cache to pick up that task.

If sticky workflow task cannot be sync-matched, it (sometimes? usually?) means SDK is unavailable and history service will timeout the task few seconds later.
If above case happen, DB IOPS are wasted. (matching service persist the task, & maybe read it back)

Matching service should not persist sticky workflow task, by

  1. either when unable to sync match, return error and let history service retry for few times before give up
  2. or using longer sync match timeout, if unable to sync match, give up immediately
  3. change history service to dispatch task, preferable to the SDK queue (SDK which has local cache) default to normal queue; when sync match timeout for SDK queue, put the task to normal queue

NOTE: history service will timeout the sticky workflow task & create a normal workflow task, workflow will NOT be stuck

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing sticky workflow task handling across the matching service and history service, focusing on sync matching, persistence, and timeout behavior. Determine which proposed dispatch or retry approach is intended, then verify that unmatched sticky tasks avoid unnecessary database work while history service still times them out and creates a normal workflow task.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.