temporalio / temporalio/temporal
disable XDC cache when not using multi-cluster replication
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 23.2k
- Forks
- 1.9k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 228
Description
Is your feature request related to a problem? Please describe.
As far as I understand the XDC cache introduced by https://github.com/temporalio/temporal/pull/4379 is only pertinent when using multi-cluster replication.
This is frustrating because like the other caches in the history service, entries aren't evicted according to their TTL, so the cache keeps growing. If operators don't tune the XDC cache size limit (and it's not apparent that they should, especially if they're not using multi-cluster replication) the history service will OOM.
Describe the solution you'd like
Minimize XDC cache overheads when not using multi-cluster replication
Describe alternatives you've considered
N/A
Additional context
Ran a workload of executing a few 1000 simple workflows and the XDC cache accumulates in size (see pprof screenshot):
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the XDC cache in the history service and the cache introduced by PR #4379, then compare its behavior when multi-cluster replication is disabled. Reproduce the reported workload and inspect the pprof result. Done means the cache no longer accumulates unnecessary entries or causes avoidable overhead when multi-cluster replication is not in use.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100