temporalio / temporalio/temporal
Inconsistent tctl workflow list behaviour
Open
Nobody has claimed this yet.
potential-bug
- Dominant language
- Go
- Stars
- 23.2k
- Forks
- 1.9k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 228
Description
Expected Behavior
When repeatedly running tctl -n prd workflow list --pdt --query="ExecutionStatus=1"
I would expect that it would return the same empty list
temporal-admintools-9bccdbb6f-wt4jj:/etc/temporal$ tctl --ns prd workflow list --pdt --query="ExecutionStatus=1"
WORKFLOW TYPE | WORKFLOW ID | RUN ID | TASK QUEUE | START TIME | EXECUTION TIME | END TIME
temporal-admintools-9bccdbb6f-wt4jj:/etc/temporal$ tctl --ns prd workflow list --pdt --query="ExecutionStatus=1"
WORKFLOW TYPE | WORKFLOW ID | RUN ID | TASK QUEUE | START TIME | EXECUTION TIME | END TIME
create-dce-key | fb8c5155-25a6-44a0-a9d4-cfb3c177961b | b0a74e13-3274-4f5b-9e85-af70fec2a718 | prd.keycustodian.command | 2024-01-29T11:03:14Z | 2024-01-29T11:03:14Z | 0001-01-01T00:00:00Z
mutex-workflow | mutex-id-piedpiper::6c67b887-b967-3e01-adab-19f07885d6ed | 6cd24d0b-5fa1-4420-9f18-9127b590c7ab | prd.keycustodian.command | 2024-01-29T11:03:13Z | 2024-01-29T11:03:13Z | 0001-01-01T00:00:00Z
vault-commands | prd.attribute.pv-xQ/wsEgMSyL4pTEumHs7nTpNrHI8Rd1LdpfqyK8DynQ= | eb31859e-2ac8-47ae-8f14-0718d4957d5c | prd.attribute.commands | 2024-01-29T11:03:13Z | 2024-01-29T11:03:13Z | 0001-01-01T00:00:00Z
mutex-workflow | mutex-id-piedpiper::ca1f5c5b-2907-3b75-b08d-1b4cdcdbcdd8 | a1c72520-ad82-4c1e-b706-986f981af3a9 | prd.keycustodian.command | 2024-01-29T11:03:13Z | 2024-01-29T11:03:13Z | 0001-01-01T00:00:00Z
create-dce-key | c7cab2ef-1fd9-4e6a-9b9f-3b778d588be6 | a8d26afa-f2a4-48cc-ab70-59d13b4edc2d | prd.keycustodian.command | 2024-01-29T11:03:12Z | 2024-01-29T11:03:12Z | 0001-01-01T00:00:00Z
create-dce-key | 9e726b71-b8a0-41df-aabc-72fe767c6539 | cdd407bc-3115-479e-b74f-b36b9d5425f9 | prd.keycustodian.command | 2024-01-29T11:03:12Z | 2024-01-29T11:03:12Z | 0001-01-01T00:00:00Z
create-dce-key | 0c1e5cf6-45ae-4e4f-8aa5-70ac2be9e6d4 | 20340756-1870-4624-a052-8bf124ceaeba | prd.keycustodian.command | 2024-01-29T11:03:12Z | 2024-01-29T11:03:12Z | 0001-01-01T00:00:00Z
mutex-workflow | mutex-id-piedpiper::31f62401-6c0d-31f0-a6bb-d690ae1321a7 | 45f0729b-231b-428d-8d81-f1bc8f0c2d3c | prd.keycustodian.command | 2024-01-29T11:03:12Z | 2024-01-29T11:03:12Z | 0001-01-01T00:00:00Z
mutex-workflow | mutex-id-piedpiper::79e1a9ba-1f9e-32ea-93ae-0d9a02ede62c | 3d5409da-2911-490d-bcf7-92ea9796d896 | prd.keycustodian.command | 2024-01-29T11:03:12Z | 2024-01-29T11:03:12Z | 0001-01-01T00:00:00Z
update-search-index | vault-service:search-indexer:piedpiper:8e98a171-a0c6-338c-b311-a74c1e57378e | 79b5acd1-c6ac-4df6-b366-592e12d902d0 | prd.vault.commands | 2024-01-29T11:02:37Z | 2024-01-29T11:02:37Z | 0001-01-01T00:00:00Z
Actual Behavior
When running
tctl -n prd workflow list --pdt --query="ExecutionStatus=1"
multiple times, occasionally a set of completed WFs are returned, as shown above
Examining one of those WFs shows
temporal-admintools-9bccdbb6f-wt4jj:/etc/temporal$ tctl --ns prd workflow desc -w c7cab2ef-1fd9-4e6a-9b9f-3b778d588be6
{
"executionConfig": {
"taskQueue": {
"name": "prd.keycustodian.command",
"kind": "Normal"
},
"workflowExecutionTimeout": "0s",
"workflowRunTimeout": "0s",
"defaultWorkflowTaskTimeout": "10s"
},
"workflowExecutionInfo": {
"execution": {
"workflowId": "c7cab2ef-1fd9-4e6a-9b9f-3b778d588be6",
"runId": "a8d26afa-f2a4-48cc-ab70-59d13b4edc2d"
},
"type": {
"name": "create-dce-key"
},
"startTime": "2024-01-29T11:03:12.943155827Z",
"closeTime": "2024-01-29T11:03:17.358251094Z",
"status": "Completed",
"historyLength": "23",
"memo": {
},
"searchAttributes": {
"indexedFields": {
"BuildIds": "[\"unversioned\"]"
}
},
"autoResetPoints": {
},
"stateTransitionCount": "14"
}
}
temporal-admintools-9bccdbb6f-wt4jj:/etc/temporal$ tctl --ns prd workflow show -w c7cab2ef-1fd9-4e6a-9b9f-3b778d588be6
1 WorkflowExecutionStarted {WorkflowType:{Name:create-dce-key}, ParentInitiatedEventId:0, TaskQueue:{Name:prd.keycustodian.command, Kind:Normal},
Input:[TlBZAGOCagZoZWFkZXIwJG1hbmV0dS5wcm90b2J1Zi5SZXF1ZXN0SGVhZGVyLXJlY29yZGOEagVtYWdpYyoArM7eagd2ZXJzaW9uAGoEdHhpZGCkMWQ1MTE
zYmUtNGU1ZC00YjQ1LTkwMmYtNTlkMGM3NzhkMWVkaglwcmV2LXR4aWQiagN2aWQwHm1hbmV0dS5wcm90b2J1Zi5WYXVsdElkLXJlY29yZGODagxwcm92aWR
lci14aWRgiXBpZWRwaXBlcmoLdmF1bHQtbGFiZWxgpGNhMWY1YzViLTI5MDctM2I3NS1iMDhkLTFiNGNkY2RiY2RkOGoIa2V5c3BhY2Ui],
WorkflowExecutionTimeout:0s, WorkflowRunTimeout:0s, WorkflowTaskTimeout:10s, Initiator:Unspecified,
OriginalExecutionRunId:a8d26afa-f2a4-48cc-ab70-59d13b4edc2d, Identity:1@mcp-keycustodian-service-gw-6566847c95-dz625,
FirstExecutionRunId:a8d26afa-f2a4-48cc-ab70-59d13b4edc2d, Attempt:1, FirstWorkflowTaskBackoff:0s, ParentInitiatedEventVersion:0}
2 WorkflowTaskScheduled {TaskQueue:{Name:prd.keycustodian.command,
Kind:Normal}, StartToCloseTimeout:10s,
Attempt:1}
3 WorkflowTaskStarted {ScheduledEventId:2,
Identity:1@mcp-keycustodian-service-cp-5bc9567c9f-zj92h,
RequestId:f5823e38-1d67-4522-b70e-3de33bd7c6e5,
SuggestContinueAsNew:false, HistorySizeBytes:620}
4 WorkflowTaskCompleted {ScheduledEventId:2, StartedEventId:3,
Identity:1@mcp-keycustodian-service-cp-5bc9567c9f-zj92h,
SdkMetadata:{CoreUsedFlags:[], LangUsedFlags:[1]},
MeteringMetadata:{NonfirstLocalActivityExecutionAttempts:0}}
5 MarkerRecorded {MarkerName:LocalActivity,
Details:map{activityId:["3c7ac2f4-98ef-3d90-b7b2-4aa7ce073b8e"],
meta:[{"firstSkd":1706526193081,"atpt":1,"backoff":null}],
result:[TlBZAGoCb2s], time:[1706526193317],
type:["start-mutex-activity"]}, WorkflowTaskCompletedEventId:4}
6 WorkflowExecutionSignaled {SignalName:prd.temporal.mutex/acquired,
Input:[TlBZAGOBag5hY3F1aXNpdGlvbi1pZGCkZjY2OGRiNzQtMDNkZS0zOTUyLTg0OTYtMDg4Yzg1NGNhNmUw],
Identity:history-service}
7 WorkflowTaskScheduled {TaskQueue:{Name:1@mcp-keycustodian-service-cp-5bc9567c9f-zj92h:502dcdd9-3d81-4828-ad63-8e99601c01dd,
Kind:Sticky}, StartToCloseTimeout:10s, Attempt:1}
8 WorkflowTaskStarted {ScheduledEventId:7,
Identity:1@mcp-keycustodian-service-cp-5bc9567c9f-zj92h,
RequestId:95864549-cdb7-4e72-ba25-f7649073b975,
SuggestContinueAsNew:false, HistorySizeBytes:1526}
9 WorkflowTaskCompleted {ScheduledEventId:7, StartedEventId:8,
Identity:1@mcp-keycustodian-service-cp-5bc9567c9f-zj92h,
MeteringMetadata:{NonfirstLocalActivityExecutionAttempts:0}}
10 MarkerRecorded {MarkerName:LocalActivity,
Details:map{activityId:["ce4d139a-1f8a-30b6-98ea-5e6f56d71ec3"],
meta:[{"firstSkd":1706526195131,"atpt":1,"backoff":null}],
result:[TlBZAAk], time:[1706526194555], type:["dce-key-exists?"]},
WorkflowTaskCompletedEventId:9}
11 MarkerRecorded {MarkerName:SideEffect,
Details:map{data:[TlBZAA8AlQTMpL6T6/fJmNVZ/BnxixMoTzdmpUQ9aimrnq1gRdj+9+JFHNpysRgln4Ck4fGANOZdbq8pB6AeqNguk6AIVm2IedpFUSt9+YHubC
iCF5FZDBvkuUNdtUojVz11KMXF8Ij286Yehysd9l9kj1y4lzGXydWrFxqiACIdIGb0uAKznkE92XBFUZp/aIVAhPOlL428ZSje]}, WorkflowTaskCompletedEventId:9}
12 ActivityTaskScheduled {ActivityId:76b7d770-0ed8-3fa7-8fc7-2623d6ffe472, ActivityType:{Name:create-dce-record!},
TaskQueue:{Name:prd.keycustodian.command, Kind:Normal},
Input:[TlBZAGOIagR0eGlkYKQxZDUxMTNiZS00ZTVkLTRiNDUtOTAyZi01OWQwYzc3OGQxZWRqA3VpZGCvcGllZHBpcGVyOjpjYTFmNWM1Yi0yOTA3LTNiNzUtYjA
4ZC0xYjRjZGNkYmNkZDhqBXJlYWxtYIlwaWVkcGlwZXJqBmtzcGFjZSJqBWxhYmVsYKRjYTFmNWM1Yi0yOTA3LTNiNzUtYjA4ZC0xYjRjZGNkYmNkZDhqA2t
leQ8AlQTMpL6T6/fJmNVZ/BnxixMoTzdmpUQ9aimrnq1gRdj+9+JFHNpysRgln4Ck4fGANOZdbq8pB6AeqNguk6AIVm2IedpFUSt9+YHubCiCF5FZDBvkuUN
dtUojVz11KMXF8Ij286Yehysd9l9kj1y4lzGXydWrFxqiACIdIGb0uAKznkE92XBFUZp/aIVAhPOlL428ZSjeagVzaGFyZCoAAAARag1sYXN0X3JvdGF0aW9
uWgAAAAAAAAAA], ScheduleToCloseTimeout:0s, ScheduleToStartTimeout:30s, StartToCloseTimeout:10s, HeartbeatTimeout:0s,
WorkflowTaskCompletedEventId:9, RetryPolicy:{InitialInterval:1s, BackoffCoefficient:2, MaximumInterval:1m40s, MaximumAttempts:3,
NonRetryableErrorTypes:[]}}
13 ActivityTaskStarted {ScheduledEventId:12,
Identity:1@mcp-keycustodian-service-cp-5bc9567c9f-zj92h,
RequestId:a4546f5a-1502-401c-9953-0877cb466349,
Attempt:1}
14 ActivityTaskCompleted {Result:[TlBZAGoHY3JlYXRlZA],
ScheduledEventId:12, StartedEventId:13,
Identity:1@mcp-keycustodian-service-cp-5bc9567c9f-zj92h}
15 WorkflowTaskScheduled {TaskQueue:{Name:1@mcp-keycustodian-service-cp-5bc9567c9f-zj92h:502dcdd9-3d81-4828-ad63-8e99601c01dd,
Kind:Sticky}, StartToCloseTimeout:10s, Attempt:1}
16 WorkflowTaskStarted {ScheduledEventId:15,
Identity:1@mcp-keycustodian-service-cp-5bc9567c9f-zj92h,
RequestId:244027ee-1e7a-46e0-80e7-532beb216d1c,
SuggestContinueAsNew:false, HistorySizeBytes:3293}
17 WorkflowTaskCompleted {ScheduledEventId:15, StartedEventId:16,
Identity:1@mcp-keycustodian-service-cp-5bc9567c9f-zj92h,
MeteringMetadata:{NonfirstLocalActivityExecutionAttempts:0}}
18 SignalExternalWorkflowExecutionInitiated {WorkflowTaskCompletedEventId:17, NamespaceId:556fb5b4-1af3-442d-ad66-66617cca5380,
WorkflowExecution:{WorkflowId:mutex-id-piedpiper::ca1f5c5b-2907-3b75-b08d-1b4cdcdbcdd8},
SignalName:mutex-release-c7cab2ef-1fd9-4e6a-9b9f-3b778d588be6,
Input:[TlBZAGOCaghpbnN0YW5jZWOBag5hY3F1aXNpdGlvbi1pZGCkZjY2OGRiNzQtMDNkZS0zOTUyLTg0OTYtMDg4Yzg1NGNhNmUwagR0eXBlagdyZWxlYXNl],
ChildWorkflowOnly:false}
19 ExternalWorkflowExecutionSignaled {InitiatedEventId:18, Namespace:prd,
NamespaceId:556fb5b4-1af3-442d-ad66-66617cca5380,
WorkflowExecution:{WorkflowId:mutex-id-piedpiper::ca1f5c5b-2907-3b75-b08d-1b4cdcdbcdd8}}
20 WorkflowTaskScheduled {TaskQueue:{Name:1@mcp-keycustodian-service-cp-5bc9567c9f-zj92h:502dcdd9-3d81-4828-ad63-8e99601c01dd,
Kind:Sticky}, StartToCloseTimeout:10s, Attempt:1}
21 WorkflowTaskStarted {ScheduledEventId:20,
Identity:1@mcp-keycustodian-service-cp-5bc9567c9f-zj92h,
RequestId:cc8b085e-7c6f-462b-9984-cfa6bb811ad8,
SuggestContinueAsNew:false, HistorySizeBytes:4086}
22 WorkflowTaskCompleted {ScheduledEventId:20, StartedEventId:21,
Identity:1@mcp-keycustodian-service-cp-5bc9567c9f-zj92h,
MeteringMetadata:{NonfirstLocalActivityExecutionAttempts:0}}
23 WorkflowExecutionCompleted {Result:[TlBZAHFqAm9rEw],
WorkflowTaskCompletedEventId:22}
Steps to Reproduce the Problem
- run workflow list multiple times ... occasionally seeing a set of apparently completed workflows
tctl --ns prd workflow list --pdt --query="ExecutionStatus=1"
Specifications
- Version: 1.22.4
- Platform: kubernetes without elasticsearch extended visibility
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the repeated tctl --ns prd workflow list --pdt --query="ExecutionStatus=1" calls and compare the returned workflows with workflow desc and workflow show. Trace the tctl workflow list entry point and its visibility query handling for --pdt and ExecutionStatus=1; done means completed workflows are not intermittently returned for this query.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, cli
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100