Retry sending notification when it fails, every couple of minutes
Nobody has claimed this yet.
- Dominant language
- PLpgSQL
- Stars
- 1.6k
- Forks
- 358
- Avg merge
- 20h 30m
- Merged PRs (30d)
- 27
Description
When sending the notification fails for whatever reason, We don't retry sending it again until the hydra-queue-runner service is restarted. Thus leading to batch of notifications when we restart it. This can be very confusing since the service can be working for couple of weeks if not months. I suggest for now to add a flag to mark the notification as failed and when running the queue-runner we only care about those that didn't fail. I don't think that anyone is still interested. Or we can retry sending failed ones every couple of minutes.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at src/hydra-queue-runner/hydra-queue-runner.cc around line 850, where notification sending currently requires a service restart after failure. Trace how pending notifications are selected and determine the retry behavior and failure state to implement; done means failed notifications are retried periodically without restarting hydra-queue-runner.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, ci-cd
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100