LightEpoch: waiterSemaphore count grows without bound under epoch-table contention → SemaphoreFullException kills every session and permanently stops the expiry/collect tasks (v1.0.98 → v2.1.7)

### Describe the bug

Describe the bug

LightEpoch.Release() signals waiterSemaphore on every epoch release while any thread is waiting for a table entry, but each waiter consumes exactly one signal. Under sustained epoch-table exhaustion the semaphore's CurrentCount climbs without bound. waiterSemaphore is constructed as new SemaphoreSlim(0) (maxCount = int.MaxValue), so after 2,147,483,647 unconsumed signals SemaphoreSlim.Release() throws SemaphoreFullException.

From that moment, every Release() executed while waiterCount > 0 throws. The exception propagates out of whatever holds the epoch:

client sessions die in RespServerSession.ProcessMessages (and their Dispose() throws again, because StorageSession.Dispose() → LightEpoch.SuspendResume() needs an epoch release too);
the background ObjectCollectTaskAsync and ExpiredKeyDeletionScanTaskAsync die with "won't be resumed" and never run again.

Nothing decrements the count except waiters consuming one signal each, so the server never recovers. Only a process restart clears it.

The signaling logic was introduced in #1543 (v1.0.98). The Release() / ReserveEntryWait() / waiterSemaphore code is unchanged in v2.1.7 (current latest), so all versions from v1.0.98 onward are affected. #2015 (v2.1.3) hardens a different aspect of LightEpoch and does not touch this path.

Environment
Garnet 1.1.8, standalone, Windows Server 2022, .NET 10.0.8, x64, 24 logical processors → kTableSize = max(128, 24*2) = 128
Non-default options: MemorySize 5g, PageSize 4m, IndexSize 64m, IndexMaxSize 1g, ObjectStoreHeapMemorySize 1g, SegmentSize 16m, ObjectStoreSegmentSize 16m, EnableStorageTier false, CompactionType None, ExpiredKeyDeletionScanFrequencySecs 120, ExpiredObjectCollectionFrequencySecs 60, LogLevel Information
Workload: ~35 client processes (StackExchange.Redis) driving a multi-million-key file-processing pipeline; mixed SET/GET/HSET/HGET/HDEL/HEXISTS/RPUSH/LINDEX/LRANGE/LREM/INFO
What we observed

Garnet ran clean for ~47 hours after start, then:

[2026-09-10 07:40:50.5731] (Critical) [127.0.0.1:54366] ProcessMessages threw an exception:
System.Threading.SemaphoreFullException: Adding the specified count to the semaphore would cause it to exceed its maximum count.
at System.Threading.SemaphoreSlim.Release(Int32 releaseCount)
at Garnet.server.GarnetApi`3.SET(ArgSlice key, ArgSlice value) in /_/libs/server/API/GarnetApi.cs:line 172
at Garnet.server.RespServerSession.NetworkSET[TGarnetApi](TGarnetApi& storageApi) in /_/libs/server/Resp/BasicCommands.cs:line 287
at Garnet.server.RespServerSession.ProcessMessages() in /_/libs/server/Resp/RespServerSession.cs:line 639
at Garnet.server.RespServerSession.TryConsumeMessages(Byte* reqBuffer, Int32 bytesReceived) in /_/libs/server/Resp/RespServerSession.cs:line 484

(LightEpoch.Release() is AggressiveInlining, so it is usually elided from the trace; it appears explicitly on the SuspendResume path.)

The onset is a synchronized burst, not a sporadic failure. In the first second after the first throw, 1,943 sessions threw the same exception — 13 of them in the same 100 µs tick, 8 in the next, 9 in the next. That is what a shared counter crossing a threshold looks like; a double-release race would produce isolated failures.

Within 2.4 s both background tasks were gone for good:

[2026-09-10 07:40:51.9944] (Critical) Unknown exception received for background hash collect task. Object collect task won't be resumed.
System.Threading.SemaphoreFullException: Adding the specified count to the semaphore would cause it to exceed its maximum count.
at System.Threading.SemaphoreSlim.Release(Int32 releaseCount)
at Tsavorite.core.ClientSession`8.Dispose() in /_/libs/storage/Tsavorite/cs/src/core/ClientSession/ClientSession.cs:line 133
at Tsavorite.core.AllocatorBase`4.ScanLookup[...]
at Tsavorite.core.GenericAllocatorImpl`3.ScanCursor[...]
at Garnet.server.DatabaseManagerBase.ExecuteHashCollect(StorageSession storageSession) in /_/libs/server/Databases/DatabaseManagerBase.cs:line 710
at Garnet.server.StoreWrapper.ObjectCollectTaskAsync(Int32 objectCollectFrequencySecs, CancellationToken token) in /_/libs/server/StoreWrapper.cs:line 711

[2026-09-10 07:40:52.9239] (Critical) Unknown exception received for background expired key deletion scan task. The task won't be resumed.
System.Threading.SemaphoreFullException: Adding the specified count to the semaphore would cause it to exceed its maximum count.
at System.Threading.SemaphoreSlim.Release(Int32 releaseCount)
at Tsavorite.core.LightEpoch.SuspendResume() in /_/libs/storage/Tsavorite/cs/src/core/Epochs/LightEpoch.cs:line 315
at Tsavorite.core.ClientSession`8.Dispose() in /_/libs/storage/Tsavorite/cs/src/core/ClientSession/ClientSession.cs:line 136
at Tsavorite.core.AllocatorBase`4.ScanLookup[...]
at Tsavorite.core.SpanByteAllocatorImpl`1.ScanCursor[...]
at Garnet.server.DatabaseManagerBase.MainStoreExpiredKeyDeletionScan(GarnetDatabase db) in /_/libs/server/Databases/DatabaseManagerBase.cs:line 733
at Garnet.server.StoreWrapper.ExpiredKeyDeletionScanTaskAsync(Int32 expiredKeyDeletionScanFrequencySecs, CancellationToken token) in /_/libs/server/StoreWrapper.cs:li...

Session teardown then fails too:

[2026-09-10 07:40:50.5742] (Error) Error disposing RespServerSession
System.Threading.SemaphoreFullException: ...
at System.Threading.SemaphoreSlim.Release(Int32 releaseCount)
at Tsavorite.core.LightEpoch.SuspendResume() in /_/libs/storage/Tsavorite/cs/src/core/Epochs/LightEpoch.cs:line 315
at Garnet.server.StorageSession.Dispose() in /_/libs/server/Storage/Session/StorageSession.cs:line 126
at Garnet.server.RespServerSession.Dispose() in /_/libs/server/Resp/RespServerSession.cs:line 391
at Garnet.server.GarnetServerTcp.DisposeMessageConsumer(INetworkHandler session) in /_/libs/server/Servers/GarnetServerTcp.cs:line 339

Over the following 4 days until restart: 24,622 SemaphoreFullExceptions, 18,773 ProcessMessages threw an exception, 5,847 Error disposing RespServerSession, at a steady 126–414/hour. The server kept accepting TCP connections but clients saw continuous 5000 ms timeouts and connection churn. Commands only succeeded in moments with no waiter present (when Release() skips the semaphore), so the server was intermittently functional under light load and fully unusable under heavy load. A restart fixed it immediately.

Root cause (code, v1.1.8 — identical in v2.1.7)

libs/storage/Tsavorite/cs/src/core/Epochs/LightEpoch.cs

csharp
// L98
static readonly ushort kTableSize = Math.Max((ushort)128, (ushort)(Environment.ProcessorCount * 2));

// L114, L125
readonly SemaphoreSlim waiterSemaphore = new(0); // maxCount = int.MaxValue
volatile int waiterCount = 0;

// L530–544
void Release()
{
...
(*(tableAligned + entry)).localCurrentEpoch = 0;
(*(tableAligned + entry)).threadId = 0;
entry = kInvalidIndex;
if (waiterCount > 0)
waiterSemaphore.Release(); // (1) unconditional +1 per epoch release while any waiter exists
}

// L624–653
void ReserveEntryWait(ref int entry)
{
int newCount = Interlocked.Increment(ref waiterCount);
try
{
...
while (true)
{
if (TryAcquireEntry(ref entry))
return;
waiterSemaphore.Wait(cts.Token); // (2) consumes exactly one signal per wake
}
}
finally { _ = Interlocked.Decrement(ref waiterCount); }
}

waiterSemaphore has exactly two live call sites: Release() at (1) and Wait() at (2) (Dispose() only disposes it). With W waiters present and R epoch releases in the same period, CurrentCount grows by roughly R − W. Under sustained table exhaustion every thread in the process is releasing epochs continuously while a handful wait, so R ≫ W and the count runs away. Epoch releases happen on every operation (and several times per operation through SuspendResume in retry loops), so at production rates 2³¹ is reached in hours of intermittent contention.

Once CurrentCount == int.MaxValue, SemaphoreSlim.Release() throws on every call while waiterCount > 0. Because the throw happens inside Release() after the slot has already been freed, the epoch table itself stays consistent — but the exception unwinds through whatever called Release(), i.e. the command handler or the background task.

Suggested fix

Never let the outstanding-signal count exceed the number of waiters, e.g.

csharp
if (waiterCount > 0 && waiterSemaphore.CurrentCount < waiterCount)
waiterSemaphore.Release();

or track pending signals with an Interlocked counter that is incremented in Release() only while below waiterCount and decremented when a waiter wakes. Any scheme that bounds CurrentCount to O(waiters) removes the overflow.

Separately, it would be more robust for ObjectCollectTaskAsync / ExpiredKeyDeletionScanTaskAsync to log and continue (or restart with backoff) on an unexpected exception rather than terminating permanently — today one transient failure disables expiry for the lifetime of the process.

Steps to reproduce

We have not built a minimal repro; the mechanism is visible from the code. A direct test would be: create a LightEpoch, occupy all kTableSize entries from kTableSize threads that hold their epoch, start one more thread that blocks in ReserveEntryWait, then have the holders loop SuspendResume() while the waiter stays blocked. Observe waiterSemaphore.CurrentCount growing monotonically — with the field made observable, the growth is immediate; the throw follows once it reaches int.MaxValue.

In production the trigger is simply sustained concurrency above kTableSize (128 on ≤64-core machines), which a large multi-client workload reaches routinely.

Expected behavior

Release() should never throw, and epoch-table contention should degrade into waiting, not into permanent server failure.

Additional context
Introduced by #1543 (v1.0.98, 2026-02-12), which replaced the previous spin-wait with this semaphore backoff.
Code path unchanged through v2.1.7; the #2015 hardening (v2.1.3) addresses announce-ordering only and states it does not affect production paths.
Full Garnet log (26 MB) from the affected process is available on request; excerpts above are verbatim except for elided generic type parameter lists marked [...].

### Steps to reproduce the bug

We have not built a minimal repro; the mechanism is visible from the code. A direct test would be: create a LightEpoch, occupy all kTableSize entries from kTableSize threads that hold their epoch, start one more thread that blocks in ReserveEntryWait, then have the holders loop SuspendResume() while the waiter stays blocked. Observe waiterSemaphore.CurrentCount growing monotonically — with the field made observable, the growth is immediate; the throw follows once it reaches int.MaxValue.

In production the trigger is simply sustained concurrency above kTableSize (128 on ≤64-core machines), which a large multi-client workload reaches routinely.

### Expected behavior

_No response_

### Screenshots

_No response_

### Release version

v1.1.8 (reproduced); code path unchanged through v2.1.7

### IDE

_No response_

### OS version

Windows Server 2022 (10.0.20348), .NET 10.0.8, x64, 24 logical processors

### Additional context

_No response_