redis / redis/node-redis

handling AWS Elasticache failover?

Open
#1,661 1 comment 4 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Question
Dominant language
TypeScript
Stars
17.6k
Forks
2k
Avg merge
2d 3h
Merged PRs (30d)
40

Description

Hi, I am investigating how to configure Node.Redis (3.1.2) to handle an Elasticache failover (for example when we do a rolling update of settings on our cluster). I specified a 'options.retry_strategy' to return a non-zero 'delay' duration, but for some reason our connection still gets destroyed before our retry_strategy even gets called. When this happens, the 'options' parameter that gets passed into our retry_strategy callback is this:
{"attempt":1,"error":null,"total_retry_time":0,"times_connected":1}

I'm aware of the 'enable_offline_queue' option but I see that the default value is 'true' so I'm not specifying it explicitly.

I'm also aware of the 'retry_unfulfilled_commands' option but it comes with this warning:
"Use this with caution if you use state altering commands (e.g. incr). This is especially useful if you use blocking commands."
I'm nervous about using this in a production environment without any additional information.

So is there a combination of flags/options that will detect that a failover happened and buffer redis commands until a connection to a backup is established?

Thanks in advance,
Jeff

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the Node.Redis 3.1.2 retry_strategy, enable_offline_queue, and retry_unfulfilled_commands documentation, then trace the connection-failure path during an ElastiCache failover. Reproduce the reported rolling-update scenario and determine whether the requested buffering and reconnection behavior is supported; done means the behavior and safe option combination are clearly established or specified for implementation.

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js, redis
Domain
backend, databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.