temporalio / temporalio/temporal

Refinement of ResourceExhausted reasons

Open
#5,352 0 comments 0 reactions 1 assignee View on GitHub

@tdeebswihart is already working on this.

Since Jan 26, 2024.

enhancement
Dominant language
Go
Stars
23.2k
Forks
1.9k
Avg merge
2d 8h
Merged PRs (30d)
228

Description

Is your feature request related to a problem? Please describe.
Currently, there are three ResourceExhausted errors with the RESOURCE_EXHAUSTED_CAUSE_BUSY_WORKFLOW reason.

  1. ErrWorkflowClosing, in my opinion, does not necessarily mean exhaustion.
    https://github.com/temporalio/temporal/blob/aec2c100b6fae91a53f0f4ea3befe8ea683f0d9c/service/history/consts/const.go#L71-L72

  2. ErrConsistentQueryBufferExceeded indicates that there are too many queries but BUSY_WORKFLOW might be too ambiguous in this case.
    https://github.com/temporalio/temporal/blob/aec2c100b6fae91a53f0f4ea3befe8ea683f0d9c/service/history/consts/const.go#L77-L78

  3. ErrResourceExhaustedBusyWorkflow explicitly includes "busy workflow" in its name. So, this error looks fine :) However, the comment indicate that this error should not be retried, but resource exhaustion usually means "retry later, please".
    https://github.com/temporalio/temporal/blob/aec2c100b6fae91a53f0f4ea3befe8ea683f0d9c/service/history/consts/const.go#L101-L102

These errors are indistinguishable in metrics, so I cannot exclude ErrWorkflowClosing (which is valid in my workload) from alerts.

Describe the solution you'd like
More granular reasons for service_errors_resource_exhausted.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.