temporalio / temporalio/temporal

RemoveSearchAttributes skips the secondary store under dual visibility

Open Beginner friendly
#12,126 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

potential-bug
Dominant language
Go
Stars
23.2k
Forks
1.9k
Avg merge
2d 8h
Merged PRs (30d)
228

Description

With dual visibility on two Elasticsearch stores, removing a custom search attribute removes it from the primary's metadata twice and never touches the secondary's. The call reports success, and the attribute reappears when the secondary is promoted.

Reproduction (v1.31.0, two ES clusters)

temporal operator search-attribute create --name DualVisRepro --type Keyword

→ cluster metadata has the attribute under both index names; mapping present on both clusters. Correct.

temporal operator search-attribute remove --name DualVisRepro --yes

→ Search attributes have been removed, exit 0. search-attribute list no longer shows it.
→ cluster metadata still has it under the secondary's index name.

Then swap visibilityStore / secondaryVisibilityStore in static config and restart the frontends (i.e. promote the secondary):

→ DualVisRepro is back in search-attribute list.

To inspect directly: SELECT encode(data,'escape') FROM cluster_metadata_info; — each index has its own attribute map in that blob.

Fix

   // operator_handler.go:385 (and :421)
   - indexName := h.visibilityMgr.GetIndexName()
   + indexName := visManager.GetIndexName()

ListSearchAttributes (:484) is worth a look too — it silently describes only the primary, which is what makes this undetectable.

Current workaround

There's no API-level option — RemoveSearchAttributesRequest carries only search_attributes and namespace, so nothing you pass can target the secondary.

If you're mid-migration, just re-run the removal after promotion. Once the old secondary becomes the primary, the same temporal operator search-attribute remove applies to it.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in operator_handler.go around lines 385 and 421, where RemoveSearchAttributes selects the visibility index, and compare the manager used there with the manager named in the issue. Inspect ListSearchAttributes around line 484 as related context, then reproduce the dual-visibility case with two Elasticsearch clusters. Done means removal updates both index names and the attribute does not reappear after promoting the secondary.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, go
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.