aws / aws/aws-cdk

(aws-elasticache): unable to make simple changes to redis cache without encountering failover error

Open
#13,389 10 comments 1 reaction 0 assignees View on GitHub
@aws-cdk/aws-elasticache bug effort/small p2 service-api
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

We are unable to make simple changes, for example editing billing tags on Redis cache nodes via CDK, when the same change is possible via the AWS Console and CLI. To allow for easy horizontal scaling of our Elasticache Redis clusters, i.e. by adding more shards, we have cluster mode enabled. As we tolerate cold caches and performance of a single node per shard is sufficient we do not deploy read replicas. When we attempt to change the billing tags via CDK we receive the error “Replication group must have at least one read replica to enable autofailover”.

### Reproduction Steps

Here is the test code (I have obscured subnets etc. in the sample)

```
import { Stack, StackProps, Construct, Tags } from "@aws-cdk/core";
import { CfnReplicationGroup } from "@aws-cdk/aws-elasticache";

export class RedisTestStack extends Stack {
constructor(scope: Construct, id: string, props?: StackProps) {
super(scope, id, props);

// Create Redis cache cluster
var redisCluster = new CfnReplicationGroup(this, "TestCluster", {
automaticFailoverEnabled: true,
cacheNodeType: "cache.t2.micro",
cacheParameterGroupName: "default.redis5.0.cluster.on",
cacheSubnetGroupName: "xxxxxxxxx",
engine: "redis",
engineVersion: "5.0.6",
numNodeGroups: 1,
replicationGroupDescription: "Test Cluster",
replicationGroupId: "TestCluster",
replicasPerNodeGroup: 0,
securityGroupIds: ["xxxxxxxx", "xxxxxxx"]
});

// Now add a tag
Tags.of(redisCluster).add('Tag1', 'Value1')
}
}
```
I can deploy the first part without issue. However when I add the Tagging code ..

```
// Now add a tag
Tags.of(redisCluster).add('Tag1', 'Value1')
```

.. and attempt to redeploy I get an error.

I can tag the redis cluster using the aws cli without any issue:

```
aws elasticache add-tags-to-resource --resource-name arn:aws:elasticache:eu-west-1:999999999999:cluster:testcluster-0001-001 --tags Key=tag2,Value=value2
```

I can also manually tag in the AWS console.

### What did you expect to happen?

I expected the cache to be tagged without service interruption similar to when using the CLI

### What actually happened?

I got the following error when I added the Tagging code and attempted to redeploy.

```
TestCluster Replication group must have at least one read replica to enable autofailover. (Service: AmazonElastiCache; Status Code: 400; Error Code: InvalidReplicationGroupState;
Request ID: d74aa3d1-4ef5-472b-8eca-e5fdb508a7d4; Proxy: null)
```

This issue is not identified with synth or in diff.

### Environment

I have tried with CDK 1.73.0 and 1.90.1 - Same result.

Here is the package.json file from the test

```
{
"name": "redis_test",
"version": "0.1.0",
"bin": {
"redis_test": "bin/redis_test.js"
},
"scripts": {
"build": "tsc",
"watch": "tsc -w",
"test": "jest",
"cdk": "cdk"
},
"devDependencies": {
"@aws-cdk/assert": "1.73.0",
"@types/jest": "^26.0.10",
"@types/node": "10.17.27",
"aws-cdk": "1.73.0",
"jest": "^26.4.2",
"ts-jest": "^26.2.0",
"ts-node": "^8.1.0",
"typescript": "~3.9.7"
},
"dependencies": {
"@aws-cdk/aws-elasticache": "1.90.1",
"@aws-cdk/core": "1.90.1",
"source-map-support": "^0.5.16"
}
}
```

### Other

I would like to provision a Redis cache with cluster mode enabled so we can modify the number of shards if later required. That was straightforward to deploy initially as a single shard. However I then modified the CDK to tag the Redis cache with tags. This change failed to apply, complaining that it was unable to failover (only one node in the replication group). I could resolve this issue by adding read replicas to each shard but the function of this cache does not require a failover node and the additional cost is then undesirable. Tagging the node similarly outside of CDK does not require read replicas or for the node to be taken out of service.

I experimented with a Redis cache (Cluster mode disabled). I can tag the cache with a subsequent modification to the CDK project. However I cannot then add shards and, in the event I needed to scale, I would need to destroy the cache and create as cluster mode enabled. This is again not ideal.

I understand that a single shard without a read replica cannot support failover but it is not clear why CDK code demands failover for tagging the cache. Within the AWS console I can easily tag the nodes without any interruption to service.

If you could please advise what CDK configuration I am missing to make this possible that would be appreciated.

---

This is :bug: Bug Report

Contributor guide

Open the contributing guide

Research direction

Start with the CfnReplicationGroup definition and the Tags.of(redisCluster).add call shown in the reproduction, then compare the synthesized template or diff with the deployment request. Check how the AWS CDK 1.73.0 and 1.90.1 versions represent tagging for ElastiCache replication groups. Done means adding a tag works for cluster mode with replicasPerNodeGroup set to 0, without triggering the autofailover error.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.