Shopify / Shopify/ci-queue

Memory leak from Redis due to missing key expirations

Open
#280 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Ruby
Stars
239
Forks
36
Avg merge
28m
Merged PRs (30d)
3

Description

Hi, we are using ci-queue for minitest and noticed that the memory usage in our Redis cluster grows slowly over time, despite using the default TTL of 8 hours. It appears that this is because not all keys have a corresponding expire call. For example, the requeues-count key is never expired. Same thing for the running key. Is there any reason that only certain keys are marked as expired?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing where the Redis keys requeues-count and running are created and where the default 8-hour TTL is applied. Verify which queue keys receive expiration and inspect the linked code search for these keys; done means the affected keys no longer persist indefinitely and the behavior is covered by the project's existing tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
redis, ruby
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.