temporalio / temporalio/temporal

Invalid system.secondaryVisibilityWritingMode value silently fails every visibility write

Open
#12,109 1 comment 0 reactions 1 assignee View on GitHub

@rodrigozhou is already working on this.

Since Sep 18, 2026.

enhancement
Dominant language
Go
Stars
23.2k
Forks
1.9k
Avg merge
2d 8h
Merged PRs (30d)
228

Description

system.secondaryVisibilityWritingMode accepts exactly three values — off, on, dual

So a typo — "duel", or "dual " with a trailing space — is accepted everywhere it is checked, and only fails at write time, on every write, in defaultManagerSelector.writeManagers()

Since this is a cluster wide global setting, a small typo currently can cause a lot of harm:

  • It can end up in visibility tasks being sent to DLQ The error is a serviceerror.Internal. With the default history.TaskDLQInternalErrors=false it is not treated as terminal, so the task retries as an unexpected error — and at history.TaskDLQUnexpectedErrorAttempts (default 70, roughly 70 minutes) the visibility task is sent to the DLQ.
    Users would need to use tdbg workflow refresh tasks to rebuild which is a pain.

  • Symptom does not point at the cause. Workflow execution is unaffected and visibility reads keep working normally, so the cluster looks healthy. No real way to alert on this, and server starts and operates fine.

Ask:

  • Validate the value where it is loaded, so a bad value is reported by validate-dynamic-config and logged as a warning at startup.
  • Treat an unrecognized value as off and log a warning, so a typo degrades to "primary only" instead of "no visibility writes at all". This keeps the cluster functional and is recoverable without a DLQ replay.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.