Retry interval is not always respected

Expected Behavior

With following retry policy, we expect 60 attempts with 1min between each attempt (assuming the activity always hits a retriable error)

temporal.RetryPolicy{
	InitialInterval:        time.Minute,
	BackoffCoefficient:     1,
	MaximumAttempts:        60,
}

Actual Behavior

Some attempts have less than 1 minute between them. In the logs below, attempts 6, 17, and 53 occur less than 1 second after the log for the preceding event.

2025/03/21 19:25:39 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-nvgd9@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 1 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 19:26:40 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-8m4x8@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 2 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 19:27:40 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-9vkgw@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 3 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 19:28:41 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-nvgd9@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 4 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 19:29:41 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-nvgd9@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 5 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 19:29:42 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-9vkgw@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 6 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 19:30:42 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-nvgd9@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 7 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 19:31:43 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-nvgd9@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 8 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 19:32:43 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-9vkgw@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 9 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 19:33:44 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-8m4x8@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 10 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 19:34:45 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-nvgd9@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 11 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 19:35:45 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-8m4x8@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 12 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 19:36:46 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-9vkgw@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 13 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 19:37:46 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-9vkgw@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 14 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 19:38:47 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-8m4x8@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 15 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 19:39:47 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-8m4x8@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 16 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 19:39:48 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-nvgd9@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 17 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 19:40:48 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-9vkgw@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 18 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 19:41:49 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-nvgd9@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 19 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 19:42:49 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-8m4x8@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 20 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 19:43:50 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-8m4x8@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 21 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 19:44:51 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-nvgd9@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 22 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 19:45:51 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-8m4x8@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 23 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 19:46:52 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-nvgd9@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 24 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 19:47:53 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-8m4x8@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 25 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 19:48:53 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-9vkgw@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 26 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 19:49:54 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-8m4x8@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 27 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 19:50:54 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-nvgd9@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 28 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 19:51:55 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-9vkgw@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 29 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 19:52:56 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-9vkgw@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 30 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 19:53:56 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-8m4x8@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 31 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 19:54:56 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-nvgd9@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 32 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 19:55:57 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-9vkgw@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 33 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 19:56:57 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-9vkgw@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 34 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 19:57:58 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-9vkgw@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 35 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 19:58:58 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-nvgd9@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 36 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 19:59:59 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-8m4x8@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 37 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 20:00:59 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-9vkgw@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 38 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 20:02:00 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-nvgd9@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 39 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 20:03:00 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-8m4x8@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 40 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 20:04:01 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-9vkgw@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 41 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 20:05:01 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-8m4x8@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 42 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 20:06:02 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-9vkgw@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 43 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 20:07:02 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-8m4x8@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 44 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 20:08:03 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-nvgd9@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 45 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 20:09:04 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-9vkgw@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 46 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 20:10:04 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-8m4x8@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 47 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 20:11:04 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-9vkgw@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 48 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 20:12:05 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-8m4x8@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 49 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 20:13:06 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-8m4x8@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 50 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 20:14:06 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-nvgd9@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 51 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 20:15:07 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-nvgd9@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 52 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 20:15:08 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-9vkgw@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 53 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 20:16:08 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-8m4x8@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 54 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 20:17:09 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-nvgd9@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 55 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 20:18:10 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-9vkgw@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 56 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 20:19:11 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-9vkgw@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 57 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 20:20:11 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-9vkgw@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 58 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 20:21:11 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-8m4x8@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 59 Error otdr not complete, need to retry (type: RetriableError, retryable: true)
2025/03/21 20:22:12 ERROR Activity error. Namespace temporal-system TaskQueue perm-task-queue WorkerID 1@azfiber-workflow-worker-deployment-79fc667848-nvgd9@ WorkflowID RunFiberCutWorkflow-1742584915037-test RunID a701d998-0606-4443-b0fc-5dc9dfe02fe2 ActivityType RunPollOtdr Attempt 60 Error otdr not complete, need to retry (type: RetriableError, retryable: true)

Steps to Reproduce the Problem

I don't have an exact repro, but we are running with 3 replicas of the temporal server and 3 replicas of our worker. That seems to be relevant as I am unable to repro locally using the temporal server shipped with the CLI.

Specifications

• Version: 1.26.2
• Platform: