lablup / lablup/backend.ai

Remove the HiRedisQueue message queue implementation

Open
#13,745 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

### Goal

Drop HiRedisQueue and keep RedisQueue as the only message queue implementation. It has stayed behind an off-by-default experimental flag since 25.12.0, and maintaining a second transport means every message queue change has to be written and verified twice.

### Work

- Delete src/ai/backend/common/message_queue/hiredis_queue.py and src/ai/backend/common/redis_client.py - the latter exists only to serve HiRedisQueue and has no other importer.
- Remove the selection branch at each call site so RedisQueue.create() is used unconditionally: manager/dependencies/messaging/message_queue.py, agent/agent.py, storage/server.py, appproxy/coordinator/server.py.
- Remove the use_experimental_redis_event_dispatcher config field from the manager, agent, storage-proxy and appproxy-coordinator unified configs, and from any sample or deployment config that sets it. A config file still carrying the key must not fail to load.
- Drop the direct hiredis requirement from requirements.txt. Keep redis[hiredis], which pulls it in as a parser extra.
- Update message_queue AGENTS.md, which currently describes hiredis_queue.py as legacy.

### Done when

- No reference to HiRedisQueue, RedisConnection or use_experimental_redis_event_dispatcher remains in the tree.
- Every component starts with a config that still sets the removed flag, and event flows are verified end to end on a live server.

### Notes

Anyone who did enable the flag switches transport on upgrade. The wire format is shared, so no message is lost, but the change belongs in the release notes as a breaking config change.

JIRA Issue: BA-7351

Contributor guide

Open the contributing guide

Research direction

Start by reading the listed message-queue call sites and message_queue AGENTS.md, then trace the unified configuration files and sample or deployment configs that define use_experimental_redis_event_dispatcher. Remove the HiRedisQueue-related references across the named files and verify that configurations retaining the flag load, no forbidden references remain, and event flows work end to end on a live server.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, redis
Domain
backend, distributed-systems
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.