temporalio / temporalio/temporal
Refinement of ResourceExhausted reasons
@tdeebswihart is already working on this.
Since Jan 26, 2024.
- Dominant language
- Go
- Stars
- 23.2k
- Forks
- 1.9k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 228
Description
Is your feature request related to a problem? Please describe.
Currently, there are three ResourceExhausted errors with the RESOURCE_EXHAUSTED_CAUSE_BUSY_WORKFLOW reason.
-
ErrWorkflowClosing, in my opinion, does not necessarily mean exhaustion.
https://github.com/temporalio/temporal/blob/aec2c100b6fae91a53f0f4ea3befe8ea683f0d9c/service/history/consts/const.go#L71-L72 -
ErrConsistentQueryBufferExceededindicates that there are too many queries butBUSY_WORKFLOWmight be too ambiguous in this case.
https://github.com/temporalio/temporal/blob/aec2c100b6fae91a53f0f4ea3befe8ea683f0d9c/service/history/consts/const.go#L77-L78 -
ErrResourceExhaustedBusyWorkflowexplicitly includes "busy workflow" in its name. So, this error looks fine :) However, the comment indicate that this errorshould not be retried, but resource exhaustion usually means "retry later, please".
https://github.com/temporalio/temporal/blob/aec2c100b6fae91a53f0f4ea3befe8ea683f0d9c/service/history/consts/const.go#L101-L102
These errors are indistinguishable in metrics, so I cannot exclude ErrWorkflowClosing (which is valid in my workload) from alerts.
Describe the solution you'd like
More granular reasons for service_errors_resource_exhausted.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.