temporalio / temporalio/temporal

Rare test failure in CLI: "Current branch token and request branch token doesn't match."

Open
#6,525 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

potential-bug teams/cgs
Dominant language
Go
Stars
23.2k
Forks
1.9k
Avg merge
2d 8h
Merged PRs (30d)
228

Description

The Temporal CLI tests hit a couple rare failures in CI after updating to Server 1.25 that we (Dan and I) believe to be a Server issue—we see spurious errors of the form: Current branch token and request branch token doesn't match.

These seem very rare because neither Dan nor I were able to repro them on our local machines (despite ~100 attempts each), and when we re-ran the failing CI job, the tests succeeded.

Expected Behavior

Successful CLI test run (no such errors reported)

Actual Behavior

The following two CLI tests failed — full log output here: https://github.com/temporalio/cli/actions/runs/10885937402/job/30204717954

In TestSharedServerSuite/TestWorkflow_Reset_ReapplyExclude:

=== FAIL: temporalcli TestSharedServerSuite/TestWorkflow_Reset_ReapplyExclude (0.02s)
    commands.workflow_reset_test.go:287: 
        	Error Trace:	/home/runner/actions-runner/_work/cli/cli/temporalcli/commands.workflow_reset_test.go:287
        	Error:      	Received unexpected error:
        	            	Current branch token and request branch token doesn't match.
        	Test:       	TestSharedServerSuite/TestWorkflow_Reset_ReapplyExclude

In TestSharedServerSuite/TestActivity_Complete:

=== FAIL: temporalcli TestSharedServerSuite/TestActivity_Complete (0.12s)
    commands_test.go:182: Calling: activity complete --activity-id dev-activity-id --workflow-id bb9450c4-9cf0-4ee6-b0f5-c5e1c1390db5 --result "complete-activity-result" --identity MyIdentity --address 127.0.0.1:37747
    commands.activity_test.go:27: 
        	Error Trace:	/home/runner/actions-runner/_work/cli/cli/temporalcli/commands.activity_test.go:27
        	Error:      	Received unexpected error:
        	            	Current branch token and request branch token doesn't match.
        	Test:       	TestSharedServerSuite/TestActivity_Complete

TestActivity_Complete is particularly interesting because there are no workflow resets performed during this test. This test is quite simple; here's what it does:

  1. Launch a workflow
  2. Wait for the workflow to launch an activity that never completes
  3. Try to force-complete the activity using the CLI command: temporal activity complete […] --result […] --identity […]

I think this is a server-side issue because nothing in the test actually does a workflow reset, and so there should be no reason why the workflow history has diverging branches. (We are also not using clustering or MRN or anything like that; this is just a single-process dev-server, so replication doesn't come into play at all.)

Steps to Reproduce the Problem

  1. Check out temporalio/cli
  2. go test -v -count 1 -run '^TestSharedServerSuite/(TestWorkflow_Reset_ReapplyExclude|TestActivity_Complete)$' github.com/temporalio/cli/temporalcli

Specifications

  • Version: 1.25
  • Platform: ubuntu-arm

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the provided Go test command for TestSharedServerSuite/TestWorkflow_Reset_ReapplyExclude and TestActivity_Complete. Read temporalcli/commands.workflow_reset_test.go around line 287 and temporalcli/commands.activity_test.go around line 27, then inspect the linked CI log and Server 1.25 context. Done means identifying and addressing the intermittent branch-token mismatch so the affected CLI tests pass reliably.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.