redis / redis/node-redis

Kubernetes LoadBalancer doesn't work with clustering on Bitnami adaptation in Azure AKS - Redirection doesn't seem to occur

Open
#2,271 9 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Bug stale
Dominant language
TypeScript
Stars
17.6k
Forks
2k
Avg merge
2d 3h
Merged PRs (30d)
40

Description

The way Bitnami does it's Kubernetes helm setup is a little odd I have to admit. The "odd" part is the LoadBalancer cluster option.

In short, the way you connect to the cluster is through the load balancer. Because Kubernetes in general is a box blacker than the blackest black... I surmise, that what is happening is that you connect to a "single" cluster host name and then it will a join all of them together.

But that really doesn't matter being a loadbalancer or even just external IPs per cluster node. Redirection seems to be the issue here.

When I connect to the load balancer it almost works as expected and you can see it through the error here.

[2022-09-22T04:58:21.970Z] Executed 'Functions.client-test' (Failed, Id=7ddacfe5-bb7d-499a-9046-d439a7813140, Duration=272ms)
[2022-09-22T04:58:21.971Z] System.Private.CoreLib: Exception while executing function: Functions.client-test. System.Private.CoreLib: Result: Failure
Exception: MOVED 12539 <cluster pod ip 1 of 6>:6379
Stack: .

That moved is what it is supposed to be doing but it is an exception instead of just setting it to the primary node which is the <cluster pod ip 1 of 6> <<< obfuscated of course.

Here is my cluster createClient I used:

const cluster = createClient({
    url: 'redis://default<passwordHere>@<my-dns-here>.eastus.cloudapp.azure.com:6379'
});

One would think you would be able to use the createCluster method but you can't because it doesn't connect and gives me issues with the IP.

  const cluster = createCluster({
    rootNodes: [{
        url: 'redis://default:<passwordHere>@<my-dns-here>.eastus.cloudapp.azure.com:6379'
    }],
    nodeAddressMap: {
        '<cluster pod ip 1 of 6>:6379': {
          host: 'redis-stream-emqx.eastus.cloudapp.azure.com',
          port: 6379
        }
      }
  });

Not to make your eyes bleed but per the documentation this kind of makes sense to what I am trying to convey. In summary, it seems as if the redirection isn't happening when coming from the default, in this case the loadbalancer or any service that is the redis cluster i.e. clusterIP's that are of the internal cluster setup.

The replicas will be read-only replicas of the masters. By default only one service is exposed (when not using the external access mode). You will connect your client to the exposed service, regardless you need to read or write. When a write operation arrives to a replica it will redirect the client to the proper master node. For example, using redis-cli you will need to provide the -c flag for redis-cli to follow the redirection automatically.

This is what should connect and even RedisInsight connects just fine.

redis-cli -c -h $SERVICE_IP -p 6379 -a $REDIS_PASSWORD <<< In this case Service_IP is my AKS LoadBalancer

I can get it to work if I focus in on the current external IP of one of my Pods i.e. '<cluster pod ip 1 of 6> as described before but that isn't a good way of doing this and should really be the redirect happening from the -c option of connecting the cli.

This blob post illustrates the redirect perfectly as exampled:

redis-cli -c -h my-redis-redis-cluster -a $REDIS_PASSWORD
Next run four set commands from the CLI:

set one hello
set two world
set three goodbye
set four world

You will see output that indicates the client is being redirected to multiple nodes.

10.244.0.6:6379> set one hello
-> Redirected to slot [9084] located at 10.244.1.5:6379
OK
10.244.1.5:6379> set two world
-> Redirected to slot [2127] located at 10.244.0.6:6379
OK
10.244.0.6:6379> set three goodbye
-> Redirected to slot [13861] located at 10.244.2.7:6379
OK
10.244.2.7:6379> set four world
-> Redirected to slot [8296] located at 10.244.1.5:6379
OK

Environment:

  • Node.js Version: latest 16+
  • Redis Server Version: Latest Bitnami 7 debian +
  • Node Redis Version: Latest 4.3 +
  • Platform: Azure Aks Kubernetes 1.8 +

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the createClient and createCluster examples, reproducing the MOVED response through a Bitnami Redis cluster on an Azure AKS LoadBalancer. Inspect how node-redis handles cluster redirection and nodeAddressMap; done means the client follows the advertised cluster node without raising the MOVED exception.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, kubernetes, node.js, redis, typescript
Domain
cloud, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.