apache / apache/airflow

Support Airflow deployment with clustered Redis

Open
#11,580 2 comments 0 reactions 0 assignees View on GitHub
area:scheduler kind:feature provider:amazon
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

**Description**

Currently, if users try to use Redis in a clustered way (for example using AWS ElastiCache cluster with multiple nodes) they may get the following error:
```
ResponseError: CROSSSLOT Keys in request don't hash to the same slot
```
This error occurs because keys must be in the same hash slot and not just the same node. Similar problems can be also observed when running flower.

**Use case / motivation**

Support Redis in clustered mode or provide information that it is not supported.

**Related Issues**
https://stackoverflow.com/questions/45215164/airflow-crossslot-keys-in-request-dont-hash-to-the-same-slot-error-using-aws-el/48123036
https://gitter.im/apache/incubator-airflow?at=5a4fe070b48e8c3566a54db9
https://aws.amazon.com/premiumsupport/knowledge-center/elasticache-crossslot-keys-error-redis/

Contributor guide

Open the contributing guide

Research direction

The issue names no files, tests, or entry points. Start by examining Airflow and Flower's Redis usage in a clustered AWS ElastiCache setup, then determine whether clustered Redis can be supported or whether documentation should state that it is unsupported. Completion should address the reported CROSSSLOT error.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python, redis
Domain
cloud, databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.