HangfireIO / HangfireIO/Hangfire

Hangfire.Pro with Redis - Handling running out of memory?

Open
#1,014 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
10.1k
Forks
1.8k
Avg merge
1h 19m
Merged PRs (30d)
1

Description

I'm using ElasticCache for Redis... Ran a large universe job last night.

Hangfire was behaving reasonably well outside of Redis timeout issues.

Then we ran out of Redis memory and things died.

Trying to recover, we discovered:

1. There's no way to cleanly delete succeeded jobs until they have expired.
2. Hangfire keeps going until the last possible amount of memory is used.
3. Thus recovery is nearly impossible -- we tried to backup ElasticCache, but apparently it won't work if there's no memory left.

If hangfire had some detection for memory use and stopped at some point prior to disaster, or if there were an easy way to remove non-important to us data, then we could delete enough memory objects to allow a backup, and restore on a larger box.

Not expecting a short term answer to this issue, unless someone has a trick we can use right now, to get back on track.

I think we'll build our own monitoring tool, that can stop servers if this happens. One of the reasons we ran out of memory is extra logging in the console, that we can turn down.

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by reviewing Hangfire's Redis storage behavior and how succeeded-job expiration, memory usage, and console logging are handled. Done would require a defined recovery or monitoring behavior for low Redis memory, plus a way to remove non-essential data.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, redis
Domain
backend, databases, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.