temporalio / temporalio/temporal

Taskqueue scavenger emits persistence errors

Open
#1,844 4 comments 0 reactions 2 assignees View on GitHub

Nobody has claimed this yet.

potential-bug
Dominant language
Go
Stars
23.2k
Forks
1.9k
Avg merge
2d 8h
Merged PRs (30d)
228

Description

Expected Behavior

Taskqueue scavenger should run without emitting an excessive amount of persistence error metrics and not cause a dip in persistence availability.

Actual Behavior

Every time the taskqueue scavenger runs, it emits quite a few persistence_errors metrics with operations gettasks, deletetaskqueue, listtaskqueue and completetaskslessthan, enough to affect a dip in our persistence availability metric (persistence_errors vs. persistence_requests).

The scavenger does appear to be making progress and successfully deleting queues, so it is unclear what all these errors are about. Unfortunately the code does not log error messages.

Are these errors expected? Should I be excluding worker persistence metrics from my overall persistence availability metric?

Specifications

  • Version: 1.8
  • Platform:

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.