temporalio / temporalio/sdk-java

Time-skipping past activity’s startToCloseTimeout causes worker not to close

Open
#2,246 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

test server
Dominant language
Java
Stars
433
Forks
249
Avg merge
5d 6h
Merged PRs (30d)
26

Description

Expected Behavior

worker.runUntil returns shortly after workflow ends

Actual Behavior

  • worker.runUntil usually hangs, ostensibly because it fails to wait for the abandoned activity
  • Logged SDK warning:
    WARN temporal_sdk_core::worker::activities: Activity not found on completion. This may happen if the activity has already been cancelled but completed anyway. task_token=TaskToken(<activity's token>) details=Status { code: NotFound, message: "invalid activityID or activity already timed out or invoking workflow is completed", metadata: MetadataMap { headers: {"content-type": "application/grpc"} }, source: None }
    

Steps to Reproduce the Problem

Pardon my TypeScript...

Assume activity foo:

export async function foo() {
}

and workflow workflow:

const { foo } = proxyActivities<typeof activities>({ startToCloseTimeout: "60 seconds" });

export async function workflow() {
  await foo();
}

And let this be the test:

test("foo", async () => {
  testEnv = await TestWorkflowEnvironment.createTimeSkipping();
  worker = Worker.create({
    connection: testEnv.nativeConnection,
    ...
  });

  let waited = false;
  const worker = await createWorker({
    activities: {
      foo: async function() { 
        if (!waited) {
          await testEnv.sleep("61 seconds");
          waited = true;
        }
      }
    },
  });
  
  await worker.runUntil(testEnv.client.workflow.execute(workflow, {}));
  // ^ gets stuck here:
});

Specifications

  • Version: TypeScript SDK 1.11.1
  • Platform: MacOS

Discussion

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the supplied TypeScript scenario with Worker.runUntil, TestWorkflowEnvironment.createTimeSkipping, and the 60-second startToCloseTimeout. Trace the worker and activity completion behavior after the simulated 61-second sleep; done means worker.runUntil returns shortly after the workflow ends without the reported Activity not found warning.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
43/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.