stackabletech / stackabletech/superset-operator

Use Redis Sentinel for Caching/Async Queries

Open
#783 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

customer-request
Dominant language
Rust
Stars
35
Forks
4
Avg merge
12h 20m
Merged PRs (30d)
10

Description

Description

The PR https://github.com/stackabletech/superset-operator/pull/724 added celery worker support using Redis following the standard Superset example: https://superset.apache.org/admin-docs/configuration/async-queries-celery/

  • This only works for RedisCache not RedisSentinelCache
  • The schema used is always redis:// instead of sentinel://
  • The master-name cannot be configured for sentinel nodes

So Redis in HA mode is not available without overrides (not tested).

Possible untested fix

configOverrides:
  superset_config.py:
    FILE_FOOTER: |
      from flask_caching.backends.rediscache import RedisSentinelCache
      _SENTINELS = [("redis-sentinel-0.redis-sentinel-headless", 26379),
                    ("redis-sentinel-1.redis-sentinel-headless", 26379),
                    ("redis-sentinel-2.redis-sentinel-headless", 26379)]
      _MASTER = "mymaster"
      _PW = os.environ["CELERY_RESULTS_BACKEND_DATABASE_PASSWORD"]

      RESULTS_BACKEND = RedisSentinelCache(
          sentinels=_SENTINELS, master=_MASTER, password=_PW, db=0,
          key_prefix="superset_results",
          # sentinel_password=..., only if the sentinels themselves require auth
      )

      _URL = ";".join(f"sentinel://:{_PW}@{h}:{p}" for h, p in _SENTINELS) + "/0"
      class CeleryConfig(object):
          broker_url = _URL
          result_backend = _URL
          broker_transport_options = {"master_name": _MASTER}
          result_backend_transport_options = {"master_name": _MASTER}
          imports = ("superset.sql_lab", "superset.tasks.scheduler")
          worker_prefetch_multiplier = 10
          task_acks_late = True
          task_annotations = {"sql_lab.get_sql_results": {"rate_limit": "100/s"}}
      CELERY_CONFIG = CeleryConfig

This is just a workaround and will need adaptation whenever the redis replicas change.

Issue checklist

This is a simple checklist of things to bear in mind when creating a new issue.

  • Describe the use-case: As far as possible, use the pattern "As a [type of user], I would like [feature/functionality] to be able to do [specific action]." This helps identify the feature and the problem it addresses.
  • Indicate importance and urgency: Use a scale (e.g., low, medium, high) to indicate the level of importance and urgency.
  • Work-around: If there is a known work-around, describe it briefly.
  • Environment: Describe the environment where the issue occurs (e.g., SDP version, K8S version, etc.).

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the existing Redis and Celery configuration introduced by PR 724, including the superset_config.py override and CeleryConfig entry points. Compare it with RedisSentinelCache and the sentinel:// workaround shown here; done means Redis Sentinel can be configured for both caching and async-query broker/results without manual overrides, including the master name.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, redis
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.