temporalio / temporalio/temporal

History service crash due to concurrent map access when history.cacheSizeBasedLimit is enabled

Open
#10,548 2 comments 0 reactions 1 assignee View on GitHub

@prathyushpv is already working on this.

Since Jun 11, 2026.

potential-bug
Dominant language
Go
Stars
23.2k
Forks
1.9k
Avg merge
2d 8h
Merged PRs (30d)
228

Description

Expected Behavior

When history.cacheSizeBasedLimit=true, releasing a workflow context recomputes its cache size, which iterates the update registry's updates map. That map has no lock of its own — it is synchronized solely by the workflow lock — so the recompute must run while the lock is held.

Actual Behavior

The history service aborts with fatal error: concurrent map iteration and map write, because the release path drops the workflow lock before recomputing the size:

makeReleaseFunc                  // service/history/workflow/cache/cache.go
  wfContext.Unlock()             // cache.go:387  workflow lock released
  c.Release(cacheKey)            // cache.go:388  runs without the lock
    -> lru.Release               // common/cache/lru.go:274            getSize(entry.value)
       -> ContextImpl.CacheSize  // service/history/workflow/context.go:1208  updateRegistry.GetSize()
          -> registry.GetSize    // service/history/workflow/update/registry.go:485
                                  //   for key, update := range r.updates { ... }

Unlock() and Release() are separate statements, so another goroutine blocked on the same workflow lock can take it the instant Unlock() returns and write the map (r.updates[id] = upd at registry.go:234, or delete(r.updates, id) at registry.go:386) while the release path is still iterating it.
This is a runtime fatal, not a panic, so the recover() in makeReleaseFunc (cache.go:365) cannot catch it — the whole process dies.

This is a synchronization gap, not intended behavior.
The registry has no mutex of its own, and Update documents its id and request fields as accessible "only while holding workflow lock" (update.go:34) — exactly the fields GetSize reads.
The size limiter's other caller, payloadSizeLimiter (registry.go:423), runs under the lock; the release path is the sole caller that does not.
The Unlock-before-Release ordering and the size-based limit are each reasonable on their own, but together they let the size computation read lock-protected state after the lock is gone.

Suggested fix

Recompute the size in makeReleaseFunc while the lock is still held (before Unlock()), cache it on the context, and have CacheSize() return the cached value.
This removes the lock-free read entirely, so it covers both the map and the MutableState reads, and it is a no-op when the limit is off.
The cached field must be atomic, since the writer holds the workflow lock while the LRU reader holds only the cache mutex.

Steps to Reproduce the Problem

  1. Enable history.cacheSizeBasedLimit (off by default) with a non-zero cache byte limit. While off, CacheSize() returns the constant 1 (context.go:1201), so the release path never iterates the registry.
  2. Drive concurrent activity on a single workflow execution so two goroutines hold the same cached context — for example Update-with-Start, where one releases the context while the other holds the workflow lock to accept or abort an update.
  3. The history service aborts with fatal error: concurrent map iteration and map write, raised from registry.GetSize via makeReleaseFunc and lru.Release.

Specifications

  • Version: present on the latest main.
  • Component: history service.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.