temporalio / temporalio/temporal
Schedules: ScheduleWorkflowAction.VersioningOverride is stored but never applied to started workflows (V1 scheduler)
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 23.2k
- Forks
- 1.9k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 228
Description
Expected Behavior
A Schedule whose action carries a VersioningOverride (e.g. a pinned Worker
Deployment Version) should start its workflows with that override applied, the
same way a direct StartWorkflowExecution with the same override does.
This matters for schedules whose workflow input shape changes between builds: the
override is the only way to keep a schedule's declared input and the code that
receives it on the same Worker Deployment Version.
Actual Behavior
The override is accepted, persisted and readable back via DescribeSchedule, but
it is never applied to the workflows the Schedule starts. The started executions
have no versioning override at all and are routed to the Current version.
Steps to Reproduce
- Run a Temporal server and a Worker Deployment with at least two versions.
- Create a schedule whose action carries a pinned override (Go SDK):
_, err := c.ScheduleClient().Create(ctx, client.ScheduleOptions{
ID: "probe",
Spec: client.ScheduleSpec{Intervals: []client.ScheduleIntervalSpec{{Every: 30 * time.Second}}},
Action: &client.ScheduleWorkflowAction{
ID: "probe-run",
Workflow: "MyWorkflow",
TaskQueue: "my-queue",
VersioningOverride: &client.PinnedVersioningOverride{
Version: worker.WorkerDeploymentVersion{
DeploymentName: "my-deployment",
BuildID: "build-a",
},
},
},
})
temporal schedule describe --schedule-id probe -o json— the override is
stored correctly:
"pinned": {"version": {"buildId": "build-a", "deploymentName": "my-deployment"}},
"pinnedVersion": "my-deployment.build-a"
- Promote a different version (
build-b) to Current. - Let the schedule fire, then
temporal workflow describethe started run.
Observed: versioningInfo.versioningOverride is absent, and the run executed
on build-b (the Current version), not on the pinned build-a.
A second, independent signal
A direct StartWorkflowExecution carrying a pinned override for a version that
does not serve the target task queue is rejected:
Pinned version 'my-deployment:build-a' is not present in task queue 'my-queue' of type 'Workflow'
A Schedule carrying that same override, on that same task queue, is created and
fires successfully. If the Schedule forwarded the override, its start would hit
the same validation. This suggests the field is dropped rather than merely
unreported.
Control: the field is observable before any workflow task runs
A direct start with the override on a task queue the version does serve shows it
immediately, with behavior: Unspecified (no workflow task completed yet):
versioningOverride: pinned:{behavior:PINNED_OVERRIDE_BEHAVIOR_PINNED
version:{build_id:"build-a" deployment_name:"my-deployment"}}
So the absence in step 5 is a genuine absence, not an artifact of nothing having
executed yet.
Likely cause
There appear to be two scheduler implementations, and they differ on this field.
V1, service/worker/scheduler/workflow.go, builds the start request field by
field and VersioningOverride is not among them:
req := &schedulespb.StartWorkflowRequest{
Request: &workflowservice.StartWorkflowExecutionRequest{
WorkflowId: workflowID, WorkflowType: newWorkflow.WorkflowType,
TaskQueue: newWorkflow.TaskQueue, Input: newWorkflow.Input,
... RetryPolicy, Memo, SearchAttributes, Header,
LastCompletionResult, ContinuedFailure, UserMetadata, Priority,
},
}
grep -rn VersioningOverride service/worker/scheduler returns no matches.
V2, chasm/lib/scheduler/invoker_tasks.go, does forward it:
if h.config.Tweakables(scheduler.Namespace).EnableVersioningOverride {
request.VersioningOverride = requestSpec.VersioningOverride
}
with EnableVersioningOverride defaulting to true, plus a dedicated test
TestExecuteTask_ForwardsVersioningOverride.
Since history.enableCHASMSchedulerCreation defaults to false and
history.chasmSchedulerCreationRolloutPercent to 0, new schedules are created
on V1 by default, where the field is silently ignored.
Why this is easy to miss
The failure is silent and the API gives no hint: the SDK serialises the field,
the server accepts and persists it, and DescribeSchedule reads it back intact.
Nothing indicates it will not be honoured. For the input-shape use case, the
consequence is a workflow deserialising a payload written for a different build.
If this is intended to work only on the CHASM scheduler, it would help a lot to
say so in the VersioningOverride field docs (currently marked Experimental),
or to reject the field at CreateSchedule when the namespace will create the
schedule on V1.
Specifications
- Version: Server 1.31.2 (
temporalio/temporal:latest), Go SDK v1.46.0, CLI 1.7.3 - Platform: reproduced on a self-hosted dev server (Kubernetes/kind) and also
on Temporal Cloud
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in service/worker/scheduler/workflow.go, where V1 builds the StartWorkflowRequest field by field, and compare it with chasm/lib/scheduler/invoker_tasks.go. Read TestExecuteTask_ForwardsVersioningOverride and identify the corresponding V1 scheduler coverage. Done means scheduled workflow starts preserve the action's VersioningOverride and the relevant scheduler tests pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 75/100