juspay / juspay/hyperswitch

Scheduler consumer registers a new UUID consumer in the Redis group on every poll and never deletes them

Open
#13,628 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
43.7k
Forks
5.1k
Avg merge
2d 20h
Merged PRs (30d)
205

Description

The scheduler consumer creates a fresh consumer identity on every poll iteration and registers it in the Redis consumer group. Redis never expires consumer entries, so the group grows without bound for as long as the consumer runs.

## Cause

`crates/scheduler/src/consumer.rs`:

```rust
let consumer_name = format!("consumer_{}", Uuid::new_v4());
```

That is inside the polling loop, and `XREADGROUP` registers the name. Consumer entries are only ever removed by `XGROUP DELCONSUMER`.

There is already a wrapper for that — `consumer_group_delete_consumer` in `crates/redis_interface/src/commands.rs` — and it has no callers anywhere in the tree.

## Measured

On a deployment with the default 5s `loop_interval`, with no tasks flowing at all:

```
XINFO GROUPS :scheduler_stream -> consumers 142
... 20 seconds later ...
XINFO GROUPS :scheduler_stream -> consumers 146
```

One per poll, roughly 17,000 per day, indefinitely. After ~45 minutes of an idle consumer the group held 555 entries.

Nothing fails abruptly. The group metadata just grows, and the bookkeeping behind `XREADGROUP`/`XINFO`/`XAUTOCLAIM` gets progressively heavier — on a Redis that may be shared with other workloads.

## Suggested fixes

Either would do:

- Reuse a stable consumer name per process (for example derived from hostname or generated once at startup) rather than per iteration. This seems closest to the intent of a consumer group.
- Or call the existing `consumer_group_delete_consumer` after a batch completes, for consumers with no pending entries.

If it helps, the safe shape for an external sweep is to delete only consumers where `pending == 0` and idle exceeds some threshold well above the poll interval — deleting a consumer that still holds unacknowledged entries drops those tasks, which on this stream means a payment that silently never retries.

## Versions

Observed on `v1.123.1`; the code is unchanged on `main` at time of writing.

Contributor guide

Open the contributing guide

Research direction

Start in crates/scheduler/src/consumer.rs at the consumer_name creation inside the polling loop, then inspect consumer_group_delete_consumer in crates/redis_interface/src/commands.rs. Compare stable-name reuse with safe deletion of consumers that have no pending entries. Done means the Redis consumer group no longer gains an entry on every poll without risking unacknowledged tasks.

Written by the indexing model from the issue text.

Assessment

Tech stack
redis, rust
Domain
backend, databases, distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.