All certificate renewals suddenly fail. Has worked for months if not years.

After using this container for months, if not years, with minimal interaction necessary, suddenly certificates are no longer automatically updated. Nothing changed in my environment so I'm a bit at a loss what happened.

From the log:

nginx-proxy-manager-app-1  | [9/15/2022] [4:50:19 PM] [Global   ] › ℹ  info      Generating MySQL knex configuration from environment variables
nginx-proxy-manager-app-1  | [9/15/2022] [4:50:19 PM] [Global   ] › ⬤  debug     Wrote db configuration to config file: ./config/production.json
nginx-proxy-manager-app-1  | [9/15/2022] [4:50:20 PM] [Migrate  ] › ℹ  info      Current database version: 20211108145214
nginx-proxy-manager-app-1  | [9/15/2022] [4:50:20 PM] [Setup    ] › ℹ  info      Creating a new JWT key pair...
nginx-proxy-manager-app-1  | [9/15/2022] [4:50:24 PM] [Setup    ] › ℹ  info      Wrote JWT key pair to config file: /app/config/production.json
nginx-proxy-manager-app-1  | [9/15/2022] [4:50:24 PM] [Setup    ] › ℹ  info      Logrotate Timer initialized
nginx-proxy-manager-app-1  | [9/15/2022] [4:50:24 PM] [Setup    ] › ℹ  info      Logrotate completed.
nginx-proxy-manager-app-1  | [9/15/2022] [4:50:24 PM] [IP Ranges] › ℹ  info      Fetching IP Ranges from online services...
nginx-proxy-manager-app-1  | [9/15/2022] [4:50:24 PM] [IP Ranges] › ℹ  info      Fetching https://ip-ranges.amazonaws.com/ip-ranges.json
nginx-proxy-manager-app-1  | [9/15/2022] [4:50:24 PM] [IP Ranges] › ℹ  info      Fetching https://www.cloudflare.com/ips-v4
nginx-proxy-manager-app-1  | [9/15/2022] [4:50:24 PM] [IP Ranges] › ℹ  info      Fetching https://www.cloudflare.com/ips-v6
nginx-proxy-manager-app-1  | [9/15/2022] [4:50:24 PM] [SSL      ] › ℹ  info      Let's Encrypt Renewal Timer initialized
nginx-proxy-manager-app-1  | [9/15/2022] [4:50:24 PM] [SSL      ] › ℹ  info      Renewing SSL certs close to expiry...
nginx-proxy-manager-app-1  | [9/15/2022] [4:50:24 PM] [IP Ranges] › ℹ  info      IP Ranges Renewal Timer initialized
nginx-proxy-manager-app-1  | [9/15/2022] [4:50:24 PM] [Global   ] › ℹ  info      Backend PID 246 listening on port 3000 ...
nginx-proxy-manager-app-1  | [9/15/2022] [4:50:49 PM] [Express  ] › ⚠  warning   invalid signature
nginx-proxy-manager-app-1  | [9/15/2022] [4:52:43 PM] [SSL      ] › ✖  error     Error: Command failed: certbot renew --non-interactive --quiet --config "/etc/letsencrypt.ini" --preferred-challenges "dns,http" --disable-hook-validation
nginx-proxy-manager-app-1  | Failed to renew certificate npm-1 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-10 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-11 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-12 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-13 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-2 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-25 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-3 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-31 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-32 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-34 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-35 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-36 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-38 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-39 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-4 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-40 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-41 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-42 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-5 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-6 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-7 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | Failed to renew certificate npm-8 with error: urn:ietf:params:acme:error:rateLimited :: There were too many requests of a given type :: Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/failed-validation-limit/
nginx-proxy-manager-app-1  | Failed to renew certificate npm-9 with error: Some challenges have failed.
nginx-proxy-manager-app-1  | All renewals failed. The following certificates could not be renewed:
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-1/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-10/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-11/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-12/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-13/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-2/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-25/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-3/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-31/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-32/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-34/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-35/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-36/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-38/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-39/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-4/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-40/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-41/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-42/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-5/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-6/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-7/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-8/fullchain.pem (failure)
nginx-proxy-manager-app-1  |   /etc/letsencrypt/live/npm-9/fullchain.pem (failure)
nginx-proxy-manager-app-1  | 24 renew failure(s), 0 parse failure(s)
nginx-proxy-manager-app-1  |
nginx-proxy-manager-app-1  |     at ChildProcess.exithandler (node:child_process:399:12)
nginx-proxy-manager-app-1  |     at ChildProcess.emit (node:events:526:28)
nginx-proxy-manager-app-1  |     at maybeClose (node:internal/child_process:1092:16)
nginx-proxy-manager-app-1  |     at Process.ChildProcess._handle.onexit (node:internal/child_process:302:5)
nginx-proxy-manager-app-1  | `QueryBuilder#allowEager` method is deprecated. You should use `allowGraph` instead. `allowEager` method will be removed in 3.0
nginx-proxy-manager-app-1  | `QueryBuilder#eager` method is deprecated. You should use the `withGraphFetched` method instead. `eager` method will be removed in 3.0
nginx-proxy-manager-app-1  | QueryBuilder#omit is deprecated. This method will be removed in version 3.0
nginx-proxy-manager-app-1  | Model#$omit is deprected and will be removed in 3.0.
nginx-proxy-manager-app-1  | Duplicate relation "access_list" in a relation expression. You should use "a.[b, c]" instead of "[a.b, a.c]". This will cause an error in objection 2.0
nginx-proxy-manager-app-1  | [9/15/2022] [4:55:08 PM] [Nginx    ] › ℹ  info      Reloading Nginx
nginx-proxy-manager-app-1  | [9/15/2022] [4:55:30 PM] [Nginx    ] › ℹ  info      Reloading Nginx
nginx-proxy-manager-app-1  | [9/15/2022] [4:55:53 PM] [Nginx    ] › ℹ  info      Reloading Nginx
nginx-proxy-manager-app-1  | [9/15/2022] [4:55:58 PM] [SSL      ] › ℹ  info      Requesting Let'sEncrypt certificates for Cert #46: myapp.mydomain.com
nginx-proxy-manager-app-1  | [9/15/2022] [4:55:58 PM] [SSL      ] › ℹ  info      Command: certbot certonly --config "/etc/letsencrypt.ini" --cert-name "npm-46" --agree-tos --authenticator webroot --email "me@mydomain.com" --preferred-challenges "dns,http" --domains "myapp.mydomain.com"
nginx-proxy-manager-app-1  | [9/15/2022] [4:56:05 PM] [Nginx    ] › ℹ  info      Reloading Nginx
nginx-proxy-manager-app-1  | [9/15/2022] [4:56:05 PM] [Express  ] › ⚠  warning   Command failed: certbot certonly --config "/etc/letsencrypt.ini" --cert-name "npm-46" --agree-tos --authenticator webroot --email "me@mydomain.com" --preferred-challenges "dns,http" --domains "myapp.mydomain.com"
nginx-proxy-manager-app-1  | Saving debug log to /var/log/letsencrypt/letsencrypt.log
nginx-proxy-manager-app-1  | Some challenges have failed.
nginx-proxy-manager-app-1  | Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

Start with the logged certbot renew and certonly commands, the referenced /etc/letsencrypt.ini configuration, and the Let's Encrypt challenge failures. Trace the renewal path from the SSL timer and compare the failed authorization output with a successful certificate request. Done means affected certificates renew without failed challenges and the regression is covered by an appropriate project test.