queue: infinite retries after reducing max_tries and bogus retry interval
- Dominant language
- Go
- Stars
- 6.1k
- Forks
- 327
- PR merge metrics
- No merged PRs in 30d
Description
# Describe the bug
Likely to due a == comparison as opposed to >= , when a email in queue has a retry count higher than `max_retries`, it will be retried infinitely as the condition never matches:
https://github.com/foxcpp/maddy/blob/cee577790b4c6c9533c46492282d520eef420f7e/internal/target/queue/queue.go#L405
A unrelated secondary issue is seen with the retry delay; which does not match the documented formula; in fact the first 4 retry delays are the same exact 14m59.99 (see logs below). I can file a separate issue if needed.
# Steps to reproduce
This can happen in the following situation:
- an email is send, while the configuration is "max_tries = 20"
- email fails temporarily multiple times, email retry counter is now at 10
- admin reduces max_tries to 8
- email will not permfail anymore as the attemt counter is well above max_retries so a == comparison will not stop it anymore
# Log files
root@srv1:/var/lib/maddy/remote_queue# journalctl -xeu maddy.service | grep "will retry" | grep ef97411f
Feb 09 16:52:40 srv1 maddy[2732258]: queue: will retry {"attempts_count":{"support@example.net":1},"msg_id":"ef97411f","next_try_delay":"14m59.999996196s","rcpts":["support@example.net"]}
Feb 09 17:07:40 srv1 maddy[2732258]: queue: will retry {"attempts_count":{"support@example.net":2},"msg_id":"ef97411f","next_try_delay":"14m59.99999822s","rcpts":["support@example.net"]}
Feb 09 17:22:40 srv1 maddy[2732258]: queue: will retry {"attempts_count":{"support@example.net":3},"msg_id":"ef97411f","next_try_delay":"14m59.999997532s","rcpts":["support@example.net"]}
Feb 09 17:37:41 srv1 maddy[2732258]: queue: will retry {"attempts_count":{"support@example.net":4},"msg_id":"ef97411f","next_try_delay":"14m59.999996182s","rcpts":["support@example.net"]}
Feb 09 17:52:41 srv1 maddy[2732258]: queue: will retry {"attempts_count":{"support@example.net":5},"msg_id":"ef97411f","next_try_delay":"29m59.999995718s","rcpts":["support@example.net"]}
Feb 09 18:22:41 srv1 maddy[2732258]: queue: will retry {"attempts_count":{"support@example.net":6},"msg_id":"ef97411f","next_try_delay":"44m59.999996344s","rcpts":["support@example.net"]}
Feb 09 19:07:41 srv1 maddy[2732258]: queue: will retry {"attempts_count":{"support@example.net":7},"msg_id":"ef97411f","next_try_delay":"44m59.999996027s","rcpts":["support@example.net"]}
Feb 09 19:52:41 srv1 maddy[2732258]: queue: will retry {"attempts_count":{"support@example.net":8},"msg_id":"ef97411f","next_try_delay":"59m59.999996174s","rcpts":["support@example.net"]}
Feb 09 20:52:42 srv1 maddy[2732258]: queue: will retry {"attempts_count":{"support@example.net":9},"msg_id":"ef97411f","next_try_delay":"1h14m59.999997755s","rcpts":["support@example.net"]}
Feb 09 22:07:42 srv1 maddy[923]: queue: will retry {"attempts_count":{"support@example.net":10},"msg_id":"ef97411f","next_try_delay":"1h44m59.999995644s","rcpts":["support@example.net"]}
Feb 09 23:52:42 srv1 maddy[923]: queue: will retry {"attempts_count":{"support@example.net":11},"msg_id":"ef97411f","next_try_delay":"2h14m59.99999673s","rcpts":["support@example.net"]}
Feb 10 02:07:42 srv1 maddy[923]: queue: will retry {"attempts_count":{"support@example.net":12},"msg_id":"ef97411f","next_try_delay":"2h44m59.99999787s","rcpts":["support@example.net"]}
Feb 10 04:52:42 srv1 maddy[923]: queue: will retry {"attempts_count":{"support@example.net":13},"msg_id":"ef97411f","next_try_delay":"3h29m59.999997578s","rcpts":["support@example.net"]}
Feb 10 08:22:43 srv1 maddy[923]: queue: will retry {"attempts_count":{"support@example.net":14},"msg_id":"ef97411f","next_try_delay":"4h29m59.999997642s","rcpts":["support@example.net"]}
Feb 10 12:52:43 srv1 maddy[41407]: queue: will retry {"attempts_count":{"support@example.net":15},"msg_id":"ef97411f","next_try_delay":"5h29m59.999996433s","rcpts":["support@example.net"]}
Feb 10 18:22:43 srv1 maddy[41407]: queue: will retry {"attempts_count":{"support@example.net":16},"msg_id":"ef97411f","next_try_delay":"6h59m59.999997628s","rcpts":["support@example.net"]}
Feb 11 01:22:43 srv1 maddy[41407]: queue: will retry {"attempts_count":{"support@example.net":17},"msg_id":"ef97411f","next_try_delay":"8h44m59.999996512s","rcpts":["support@example.net"]}
Feb 11 10:07:43 srv1 maddy[41407]: queue: will retry {"attempts_count":{"support@example.net":18},"msg_id":"ef97411f","next_try_delay":"10h59m59.999996882s","rcpts":["support@example.net"]}
root@srv1:/var/lib/maddy/remote_queue#
When maddy was restarted (with different configuration) can be seen by the changing PID.
# Configuration file
target.queue remote_queue {
max_tries 8
target &outbound_delivery
# Environment information
* maddy version: cee577790b4c6c9533c46492282d520eef420f7e
Contributor guide
Assessment
This issue has not been assessed yet.