temporalio / temporalio/temporal

Schedules: ScheduleWorkflowAction.VersioningOverride is stored but never applied to started workflows (V1 scheduler)

Open
#12,148 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
23.2k
Forks
1.9k
Avg merge
2d 8h
Merged PRs (30d)
228

Description

Expected Behavior

A Schedule whose action carries a VersioningOverride (e.g. a pinned Worker
Deployment Version) should start its workflows with that override applied, the
same way a direct StartWorkflowExecution with the same override does.

This matters for schedules whose workflow input shape changes between builds: the
override is the only way to keep a schedule's declared input and the code that
receives it on the same Worker Deployment Version.

Actual Behavior

The override is accepted, persisted and readable back via DescribeSchedule, but
it is never applied to the workflows the Schedule starts. The started executions
have no versioning override at all and are routed to the Current version.

Steps to Reproduce
  1. Run a Temporal server and a Worker Deployment with at least two versions.
  2. Create a schedule whose action carries a pinned override (Go SDK):
_, err := c.ScheduleClient().Create(ctx, client.ScheduleOptions{
    ID:   "probe",
    Spec: client.ScheduleSpec{Intervals: []client.ScheduleIntervalSpec{{Every: 30 * time.Second}}},
    Action: &client.ScheduleWorkflowAction{
        ID:        "probe-run",
        Workflow:  "MyWorkflow",
        TaskQueue: "my-queue",
        VersioningOverride: &client.PinnedVersioningOverride{
            Version: worker.WorkerDeploymentVersion{
                DeploymentName: "my-deployment",
                BuildID:        "build-a",
            },
        },
    },
})
  1. temporal schedule describe --schedule-id probe -o json — the override is
    stored correctly:
"pinned": {"version": {"buildId": "build-a", "deploymentName": "my-deployment"}},
"pinnedVersion": "my-deployment.build-a"
  1. Promote a different version (build-b) to Current.
  2. Let the schedule fire, then temporal workflow describe the started run.

Observed: versioningInfo.versioningOverride is absent, and the run executed
on build-b (the Current version), not on the pinned build-a.

A second, independent signal

A direct StartWorkflowExecution carrying a pinned override for a version that
does not serve the target task queue is rejected:

Pinned version 'my-deployment:build-a' is not present in task queue 'my-queue' of type 'Workflow'

A Schedule carrying that same override, on that same task queue, is created and
fires successfully. If the Schedule forwarded the override, its start would hit
the same validation. This suggests the field is dropped rather than merely
unreported.

Control: the field is observable before any workflow task runs

A direct start with the override on a task queue the version does serve shows it
immediately, with behavior: Unspecified (no workflow task completed yet):

versioningOverride: pinned:{behavior:PINNED_OVERRIDE_BEHAVIOR_PINNED
  version:{build_id:"build-a" deployment_name:"my-deployment"}}

So the absence in step 5 is a genuine absence, not an artifact of nothing having
executed yet.

Likely cause

There appear to be two scheduler implementations, and they differ on this field.

V1, service/worker/scheduler/workflow.go, builds the start request field by
field and VersioningOverride is not among them:

req := &schedulespb.StartWorkflowRequest{
    Request: &workflowservice.StartWorkflowExecutionRequest{
        WorkflowId: workflowID, WorkflowType: newWorkflow.WorkflowType,
        TaskQueue: newWorkflow.TaskQueue, Input: newWorkflow.Input,
        ... RetryPolicy, Memo, SearchAttributes, Header,
        LastCompletionResult, ContinuedFailure, UserMetadata, Priority,
    },
}

grep -rn VersioningOverride service/worker/scheduler returns no matches.

V2, chasm/lib/scheduler/invoker_tasks.go, does forward it:

if h.config.Tweakables(scheduler.Namespace).EnableVersioningOverride {
    request.VersioningOverride = requestSpec.VersioningOverride
}

with EnableVersioningOverride defaulting to true, plus a dedicated test
TestExecuteTask_ForwardsVersioningOverride.

Since history.enableCHASMSchedulerCreation defaults to false and
history.chasmSchedulerCreationRolloutPercent to 0, new schedules are created
on V1 by default, where the field is silently ignored.

Why this is easy to miss

The failure is silent and the API gives no hint: the SDK serialises the field,
the server accepts and persists it, and DescribeSchedule reads it back intact.
Nothing indicates it will not be honoured. For the input-shape use case, the
consequence is a workflow deserialising a payload written for a different build.

If this is intended to work only on the CHASM scheduler, it would help a lot to
say so in the VersioningOverride field docs (currently marked Experimental),
or to reject the field at CreateSchedule when the namespace will create the
schedule on V1.

Specifications
  • Version: Server 1.31.2 (temporalio/temporal:latest), Go SDK v1.46.0, CLI 1.7.3
  • Platform: reproduced on a self-hosted dev server (Kubernetes/kind) and also
    on Temporal Cloud

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in service/worker/scheduler/workflow.go, where V1 builds the StartWorkflowRequest field by field, and compare it with chasm/lib/scheduler/invoker_tasks.go. Read TestExecuteTask_ForwardsVersioningOverride and identify the corresponding V1 scheduler coverage. Done means scheduled workflow starts preserve the action's VersioningOverride and the relevant scheduler tests pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
75/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.