temporalio / temporalio/temporal

GenerateReplicationTasks activity of ForceReplicationWorkflow should only log not found errors opposed to returning the error as well

Open
#6,468 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

potential-bug teams/cgs
Dominant language
Go
Stars
23.2k
Forks
1.9k
Avg merge
2d 8h
Merged PRs (30d)
228

Description

Expected Behavior

When generating replication tasks in GenerateReplicationTasks, not found errors from the underlying historyClient.GenerateLastHistoryReplicationTasks should only be logged.

Workflows can for example be archived between being listed and then processed in GenerateReplicationTasks. A not found history should just be logged. Returning the error keep retrying the activity

Actual Behavior

Due to returning the error the activity gets retried and does not make any progress anymore.

	for i := startIndex; i < len(request.Executions); i++ {
		we := request.Executions[i]
		if err := a.generateWorkflowReplicationTask(ctx, rateLimiter, definition.NewWorkflowKey(request.NamespaceID, we.WorkflowId, we.RunId)); err != nil {
			if !isNotFoundServiceError(err) {
				a.logger.Error("force-replication failed to generate replication task", tag.WorkflowNamespaceID(request.NamespaceID), tag.WorkflowID(we.WorkflowId), tag.WorkflowRunID(we.RunId), tag.Error(err))
				return err
			}
		}

		activity.RecordHeartbeat(ctx, i)
	}

Steps to Reproduce the Problem

  1. Execute a ForceReplicationWorkflow run for a given workflow which will archive directly once completed
  2. Complete the workflow in between ListWorkflows and GenerateReplicationTasks

Specifications

  • Version: main
  • Platform:

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in service/worker/migration/activities.go at GenerateReplicationTasks and inspect its call to historyClient.GenerateLastHistoryReplicationTasks. Verify that not-found errors are logged while processing continues, whereas other errors still stop the activity; the reproduction steps describe the completed behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.