stackabletech / stackabletech/superset-operator
Use Redis Sentinel for Caching/Async Queries
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 35
- Forks
- 4
- Avg merge
- 12h 20m
- Merged PRs (30d)
- 10
Description
Description
The PR https://github.com/stackabletech/superset-operator/pull/724 added celery worker support using Redis following the standard Superset example: https://superset.apache.org/admin-docs/configuration/async-queries-celery/
- This only works for
RedisCachenotRedisSentinelCache - The schema used is always
redis://instead ofsentinel:// - The master-name cannot be configured for sentinel nodes
So Redis in HA mode is not available without overrides (not tested).
Possible untested fix
configOverrides:
superset_config.py:
FILE_FOOTER: |
from flask_caching.backends.rediscache import RedisSentinelCache
_SENTINELS = [("redis-sentinel-0.redis-sentinel-headless", 26379),
("redis-sentinel-1.redis-sentinel-headless", 26379),
("redis-sentinel-2.redis-sentinel-headless", 26379)]
_MASTER = "mymaster"
_PW = os.environ["CELERY_RESULTS_BACKEND_DATABASE_PASSWORD"]
RESULTS_BACKEND = RedisSentinelCache(
sentinels=_SENTINELS, master=_MASTER, password=_PW, db=0,
key_prefix="superset_results",
# sentinel_password=..., only if the sentinels themselves require auth
)
_URL = ";".join(f"sentinel://:{_PW}@{h}:{p}" for h, p in _SENTINELS) + "/0"
class CeleryConfig(object):
broker_url = _URL
result_backend = _URL
broker_transport_options = {"master_name": _MASTER}
result_backend_transport_options = {"master_name": _MASTER}
imports = ("superset.sql_lab", "superset.tasks.scheduler")
worker_prefetch_multiplier = 10
task_acks_late = True
task_annotations = {"sql_lab.get_sql_results": {"rate_limit": "100/s"}}
CELERY_CONFIG = CeleryConfig
This is just a workaround and will need adaptation whenever the redis replicas change.
Issue checklist
This is a simple checklist of things to bear in mind when creating a new issue.
- Describe the use-case: As far as possible, use the pattern "As a [type of user], I would like [feature/functionality] to be able to do [specific action]." This helps identify the feature and the problem it addresses.
- Indicate importance and urgency: Use a scale (e.g., low, medium, high) to indicate the level of importance and urgency.
- Work-around: If there is a known work-around, describe it briefly.
- Environment: Describe the environment where the issue occurs (e.g., SDP version, K8S version, etc.).
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the existing Redis and Celery configuration introduced by PR 724, including the superset_config.py override and CeleryConfig entry points. Compare it with RedisSentinelCache and the sentinel:// workaround shown here; done means Redis Sentinel can be configured for both caching and async-query broker/results without manual overrides, including the master name.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, redis
- Domain
- backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100