temporalio / temporalio/temporal

Persistence rate-limit ResourceExhausted is flattened to Unavailable in ProcessOutgoingSearchAttributes

Open
#11,571 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

potential-bug
Dominant language
Go
Stars
23.2k
Forks
1.9k
Avg merge
2d 8h
Merged PRs (30d)
228

Description

Actual behaviour

service/history/api/get_history_util.go:358 converts a *serviceerror.ResourceExhausted from the
persistence rate limiter into a *serviceerror.Unavailable:

saTypeMap, err := saProvider.GetSearchAttributes(persistenceVisibilityMgr.GetIndexName(), false)
if err != nil {
      return serviceerror.NewUnavailablef(consts.ErrUnableToGetSearchAttributesMessage, err)
}

Expected behaviour

ErrPersistenceSystemLimitExceeded reaches the caller as ResourceExhausted, so the gRPC status reflects rate limiting and common/backoff/retry.go:80 applies the throttle backoff.

Impact

  • Wrong status code. Rate limiting is reported as Unavailable.
  • Throttle retry bypassed. common/backoff/retry.go:80 applies the longer backoff only to
    *serviceerror.ResourceExhausted, so retries run at the ordinary faster rate against an already exhausted persistence budget. This contradicts the intent stated at service/history/queues/executable.go:795.

Specifications

  • Version: 1.31.2
  • Platform: self-hosted postgres

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read service/history/api/get_history_util.go:358, then trace the error handling through common/backoff/retry.go:80 and service/history/queues/executable.go:795. Confirm the persistence rate-limit error remains ResourceExhausted to the caller and that the throttle backoff path is used, with regression coverage if the surrounding tests identify a suitable location.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, postgresql
Domain
backend, databases, distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.