temporalio / temporalio/temporal

Task start can remain undelivered after an ambiguous History timeout

Open
#11,733 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
23.2k
Forks
1.9k
Avg merge
2d 8h
Merged PRs (30d)
228

Description

Expected Behavior

If History commits a workflow- or activity-task start while the original worker poll is still active, Matching should be able to recover the committed response through History's existing same-request-ID contract. A per-attempt RPC deadline should not leave the task started in History but undelivered to the worker.

Actual Behavior

Matching uses a shorter child context for RecordWorkflowTaskStarted and RecordActivityTaskStarted than the parent worker long poll. History can commit the start while Matching observes the child-context deadline before receiving the response.

History already treats a repeated request ID as idempotent. However, once Matching leaves that concrete task-to-poller dispatch, a later selection creates a new request ID. History then returns TaskAlreadyStarted, Matching drops that delivery, and no worker receives the committed start response. Recovery is delayed until the workflow-task or activity timeout.

Steps to Reproduce the Problem

  1. Match a workflow or activity task to a worker poll.
  2. Allow Record*TaskStarted to commit in History, but delay its response until Matching's child context expires.
  3. Keep the original worker poll active and allow the task-start path to recover.
  4. Observe that leaving the original dispatch causes a later delivery to use a different request ID, receive TaskAlreadyStarted, and fail to deliver the committed start response.

Specifications

  • Version: current main
  • Platform: server-side behavior; platform independent

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the Matching task-start path for RecordWorkflowTaskStarted and RecordActivityTaskStarted, focusing on the child RPC deadline and request-ID handling. Reproduce the delayed History response described in the issue, then verify that a committed task start is delivered to the still-active worker poll rather than retried with a new request ID and dropped as TaskAlreadyStarted.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.