temporalio / temporalio/temporal

Allow to disable workflow deadlock detector

Open
#6,546 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Go
Stars
23.2k
Forks
1.9k
Avg merge
2d 8h
Merged PRs (30d)
228

Description

Is your feature request related to a problem? Please describe.

We need to run some function that might block for a while (up to 10s, usually) during a Temporal workflow. That function cannot receive a Temporal workflow context. It also has to run every time the workflow runs, including replays. Further operations during this workflow rely on this function being run first.

Currently when we do this, the deadlock detection mechanism panics.

Describe the solution you'd like

Allow to disable the deadlock detection for a specific scope. Possible just making https://github.com/temporalio/sdk-go/blob/master/internal/workflow_deadlock.go#L51 and its friend public.

Describe alternatives you've considered

We tried running the function in a separate go routine, and while it's running sleeping on Temporal. The problem is that this is non-deterministic, and we don't know how much Sleep we need to get.

Additional context

I initially really really really didn't want to open this issue. I really think that the deadlock detection is a Good Thing. I discussed this with @mfateev and Chad (no idea what's his alias here), and at the end came to a conclusion that there's no avoiding this.

@mfateev suggested (ab)using the DataConverterWithoutDeadlockDetection, but I'd rather do it in a more polite fashion.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with internal/workflow_deadlock.go, especially the referenced detector code, and compare it with DataConverterWithoutDeadlockDetection. Clarify the intended scope and API behavior, then verify that the selected operation can run for up to 10 seconds during workflow execution and replay without disabling detection elsewhere.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.