Bogdanp / Bogdanp/dramatiq

redis: Compatibility with AWS ElastiCache [Serverless]

Open
#754 3 comments 0 reactions 0 assignees View on GitHub
enhancement wontdo
Dominant language
Python
Stars
5.3k
Forks
383
Avg merge
9h 41m
Merged PRs (30d)
2

Description

The Redis backends in their current state do not work with AWS ElastiCache, due to certain limitations the latter imposes on the provided Redis/Valkey interfaces. This issue tracks the changes necessary to make the two cooperate happily.

This is no way about any `if AWS:` kind of hacks, but rather about finding and using the lowest common denominators between "normal" Redis and the one provided by AWS. Internally, an ElastiCache (whether "serverful" or Serverless) is a Redis Cluster, so AFAIU most restrictions actually come from the Cluster deployment model. Serverless has some further limitations on top of that, so I would propose to target compatibility with Serverless to cover everything.

AWS docs:
[Supported and restricted commands](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/SupportedCommands.html)
[Best practices for Lua scripts](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/BestPractices.Clients.Redis.LuaScripts.html)

#### Broker
- Citing [Redis docs](https://redis.io/docs/latest/develop/programmability/eval-intro/#script-parameterization):
> **Important**: to ensure the correct execution of scripts, both in standalone and clustered deployments, all names of keys that a script accesses must be explicitly provided as input key arguments. The script **should only** access keys whose names are given as input arguments. Scripts should never access keys with programmatically-generated names or based on the contents of data structures stored in the database.

AWS takes the recommendation seriously and outright disallows execution of non-conformant scripts.
- Most of the dynamic key names in `dispatch.lua` are not _that_ dynamic, we just build new keys like queue names from known static values (`namespace` etc.) - I believe this could be done in Python instead.

- `KEYS` is not supported on Serverless.
- `Lua scripts without any input keys are not supported.`
- I think this is only the case with `maxstack.lua` and is easily fixed by providing _some_ key.

#### Rate Limiter backend
- `WATCH` is not supported on Serverless, so `incr`/`decr`/`incr_and_sum` methods won't work.

This is solved by rewriting the logic of `incr`/`decr`/`incr_and_sum` as Lua scripts instead - the atomic execution would achieve the same effect without the need for `WATCH` on the client.

In fact, I've already implemented this and it's been used in production for quite a while. I can submit a PR after I tidy the code up a bit.

#### Result backend
I haven't tried using the result backend yet, but the code looks harmless enough and I expect no changes to be necessary.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.