HangfireIO / HangfireIO/Hangfire
Hangfire.Pro with Redis - Handling running out of memory?
- Dominant language
- C#
- Stars
- 10.1k
- Forks
- 1.8k
- Avg merge
- 1h 19m
- Merged PRs (30d)
- 1
Description
I'm using ElasticCache for Redis... Ran a large universe job last night.
Hangfire was behaving reasonably well outside of Redis timeout issues.
Then we ran out of Redis memory and things died.
Trying to recover, we discovered:
1. There's no way to cleanly delete succeeded jobs until they have expired.
2. Hangfire keeps going until the last possible amount of memory is used.
3. Thus recovery is nearly impossible -- we tried to backup ElasticCache, but apparently it won't work if there's no memory left.
If hangfire had some detection for memory use and stopped at some point prior to disaster, or if there were an easy way to remove non-important to us data, then we could delete enough memory objects to allow a backup, and restore on a larger box.
Not expecting a short term answer to this issue, unless someone has a trick we can use right now, to get back on track.
I think we'll build our own monitoring tool, that can stop servers if this happens. One of the reasons we ran out of memory is extra logging in the console, that we can turn down.
Contributor guide
Research direction
No files, tests, or entry points are named. Start by reviewing Hangfire's Redis storage behavior and how succeeded-job expiration, memory usage, and console logging are handled. Done would require a defined recovery or monitoring behavior for low Redis memory, plus a way to remove non-essential data.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, redis
- Domain
- backend, databases, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100