StackExchange / StackExchange/StackExchange.Redis

Timeout awaiting response with batch insertion mode

Open
#2,354 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C#
Stars
6.2k
Forks
1.6k
Avg merge
1d 15h
Merged PRs (30d)
43

Description

Hello,
I'm trying to insert around 10 millions hashsets into Redis, it works perfectly until around 6 millions than I can this timeout error.
Below is the error message and the code. Is there any improvement that I can do for the code?

StackExchange.Redis.RedisTimeoutException: Timeout awaiting response (outbound=316KiB, inbound=11774KiB, 9422ms elapsed, timeout is 5000ms), command=HMSET, next: HMSET RegistrationObject:EN14BLZ, inst: 1, qu: 3181156, qs: 0, aw: True, bw: WritingMessage, rs: DequeueResult, ws: Writing, in: 0, last-in: 2, cur-in: 215, sync-ops: 1, async-ops: 1, serverEndpoint: mkrs.redis.cache.windows.net:6380, conn-sec: 493.01, mc: 1/1/0, mgr: 8 of 10 available, clientName: (SE.Redis-v2.6.90.64945), IOCP: (Busy=0,Free=1000,Min=8,Max=1000), WORKER: (Busy=2,Free=32765,Min=8,Max=32767), POOL: (Threads=16,QueuedItems=0,CompletedItems=31687), v: 2.6.90.64945 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

            var addTasks = new List<Task>();
            IBatch batch = _database.CreateBatch();
            
            for (int i = 0; i < data.Count; i++)
            {
                var serialData = RedisExtensions.ToHashEntries(data[i]);
                Task addAsync = batch.HashSetAsync($"{typeof(TFeature).Name}:{data[i].Key}", serialData);
                addTasks.Add(addAsync);
            }
            batch.Execute();
            Task[] tasks = addTasks.ToArray();
            await Task.WhenAll(tasks);

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the batch insertion loop and the RedisTimeoutException details in the issue, then read the linked StackExchange.Redis timeouts article. Reproduce the workload with progressively smaller batches and the shown connection settings. Done means identifying whether the timeout is reproducible client behavior or workload-specific, with a documented diagnosis and actionable next steps.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, redis
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.