beanstalkd / beanstalkd/beanstalkd
What can cause a reserved job to not be ready?
- Dominant language
- C
- Stars
- 6.7k
- Forks
- 868
- PR merge metrics
- No merged PRs in 30d
Description
Ran into a strange issue that took down our application. Basically, our beanstalkd server (running the latest available version, 1.13) suddenly became unusable and we started seeing thousands of error logs that were pointing to this specific line
https://github.com/beanstalkd/beanstalkd/blob/91c54fc05dc759ef27459ce4383934e1a4f2fb4b/prot.c#L466
I believe this is the first time we've encountered this issue. Other issues we've had is we would get random segmentation or general protection faults and that seems to happen once every few weeks. I'm not sure if they are related to this new issue (I'm still hoping to capture a coredump for the segfault/general protection problem) but this is the first time we've seen this particular problem occur after years of use and it's a bit worse because it prevents usage of the queue server altogether (at least with the segfaults, our beanstalkd server process gets restarted immediately and resumes work from binlogs).
Restarting the process by hand fixed the issue immediately.
Contributor guide
Research direction
Start at prot.c line 466, which the report identifies as the source of the repeated errors. Investigate the reserved-job state associated with the reported “not ready” condition and compare behavior before and after restarting beanstalkd. Done means the triggering condition is reproducible or explained, with a focused fix or a documented cause and regression coverage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100