Consider caching parent-not-found results to avoid repeated cross-cluster RPCs in VerifyChildExecutionCompletionRecorded

Summary

service/history/api/verifychildworkflowcompletionrecorded/api.go
(line ~146) has a TODO suggesting a workflowNotFoundCache for
parent workflows that are confirmed not found:

if err != nil {
    if common.IsNotFoundError(err) {
        // parent workflow is not found on source cluster,
        // we can return empty response to indicate that verification is done
        // TODO: add parent workflow to workflowNotFoundCache
        return &historyservice.VerifyChildExecutionCompletionRecordedResponse{}, nil
    }
    ...
}

This cache does not currently exist anywhere in the codebase —
this would be a new mechanism, not wiring into an existing one.

Why this matters

This code path is reached via SyncWorkflowState (a
cross-cluster RPC call to the parent's source cluster),
invoked from transfer_queue_standby_task_executor.go when
verifying child workflow completion was recorded by the parent.

This transfer task follows the standard task-processing retry
loop. If the parent workflow genuinely no longer exists, each
retry of this task re-triggers the same cross-cluster RPC call,
just to get the same "not found" result repeatedly — until the
task eventually completes/acks.

Investigation

• Call chain: transfer_queue_standby_task_executor.go:336
  → VerifyChildExecutionCompletionRecorded (history_engine.go)
  → SyncWorkflowState (cross-cluster RPC)
• The "not found" case currently returns success immediately
  (correct behavior), but does nothing to prevent the same
  remote call from happening again on the next retry.

Open design questions

Before implementing, it would help to clarify:

1. Cache scope — per-shard, per-process, or persisted?
2. TTL / invalidation — what if a workflow with the same ID
   is later recreated (new run) after being deleted? Could a
   stale "not found" cache entry cause incorrect behavior?
3. Alternative approaches — would task-level backoff/rate-limiting
   achieve a similar effect without needing a new cache structure?

Happy to implement once the intended design is clarified, or if
there's an existing pattern elsewhere in the codebase I should
follow for this kind of cache.

References

• service/history/api/verifychildworkflowcompletionrecorded/api.go:113-147
• service/history/transfer_queue_standby_task_executor.go:320-345