temporalio / temporalio/temporal
RemoveSearchAttributes skips the secondary store under dual visibility
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 23.2k
- Forks
- 1.9k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 228
Description
With dual visibility on two Elasticsearch stores, removing a custom search attribute removes it from the primary's metadata twice and never touches the secondary's. The call reports success, and the attribute reappears when the secondary is promoted.
Reproduction (v1.31.0, two ES clusters)
temporal operator search-attribute create --name DualVisRepro --type Keyword
→ cluster metadata has the attribute under both index names; mapping present on both clusters. Correct.
temporal operator search-attribute remove --name DualVisRepro --yes
→ Search attributes have been removed, exit 0. search-attribute list no longer shows it.
→ cluster metadata still has it under the secondary's index name.
Then swap visibilityStore / secondaryVisibilityStore in static config and restart the frontends (i.e. promote the secondary):
→ DualVisRepro is back in search-attribute list.
To inspect directly: SELECT encode(data,'escape') FROM cluster_metadata_info; — each index has its own attribute map in that blob.
Fix
// operator_handler.go:385 (and :421)
- indexName := h.visibilityMgr.GetIndexName()
+ indexName := visManager.GetIndexName()
ListSearchAttributes (:484) is worth a look too — it silently describes only the primary, which is what makes this undetectable.
Current workaround
There's no API-level option — RemoveSearchAttributesRequest carries only search_attributes and namespace, so nothing you pass can target the secondary.
If you're mid-migration, just re-run the removal after promotion. Once the old secondary becomes the primary, the same temporal operator search-attribute remove applies to it.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in operator_handler.go around lines 385 and 421, where RemoveSearchAttributes selects the visibility index, and compare the manager used there with the manager named in the issue. Inspect ListSearchAttributes around line 484 as related context, then reproduce the dual-visibility case with two Elasticsearch clusters. Done means removal updates both index names and the attribute does not reappear after promoting the secondary.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, go
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100