auto-ssl / auto-ssl/lua-resty-auto-ssl

JWS Verification Error in dehydrated

Open
#202 8 comments 0 reactions 0 assignees View on GitHub
Dominant language
Lua
Stars
2k
Forks
184
PR merge metrics
No merged PRs in 30d

Description

Hey!

2 days ago I had to bump the version of this module to latest due to the ACME v1 deprecation. After bumping this module, some certificates could successfully renew in isolation, but certificates that are receiving sustained amounts of traffic appeared to be making the server crash during generation, which was causing us to hit Lets Encrypt's rate limits as the process was incomplete.

Looking in the logs, I was getting very similar log output that I mentioned here:

https://github.com/GUI/lua-resty-auto-ssl/issues/136

```
notice signal 15 (SIGTERM) received, exiting
notice signal 15 (SIGTERM) received, exiting
notice signal 14 (SIGALRM) received
notice signal 14 (SIGALRM) received
notice signal 14 (SIGALRM) received
notice signal 14 (SIGALRM) received
notice signal 14 (SIGALRM) received
notice signal 14 (SIGALRM) received
notice signal 14 (SIGALRM) received
notice signal 14 (SIGALRM) received
notice signal 14 (SIGALRM) received
notice signal 14 (SIGALRM) received
notice signal 14 (SIGALRM) received
notice signal 14 (SIGALRM) received
notice signal 17 (SIGCHLD) received
notice signal 17 (SIGCHLD) received
alert worker process 19879 exited on signal 9
alert worker process 19879 exited on signal 9
alert worker process 19879 exited on signal 9
alert shared memory zone "auto_ssl" was locked by 19879
alert shared memory zone "auto_ssl" was locked by 19879
alert shared memory zone "auto_ssl" was locked by 19879
notice exit
notice exit
INFO openresty (exit status 0)
INFO 'openresty' with pid 10560
notice using the "epoll" event method
notice using the "epoll" event method
notice openresty/1.11.2.2
notice openresty/1.11.2.2
notice built by gcc 4.9.2 (Debian 4.9.2-10)
notice built by gcc 4.9.2 (Debian 4.9.2-10)
notice OS: Linux 4.4.0-1092-aws
notice OS: Linux 4.4.0-1092-aws
notice getrlimit(RLIMIT_NOFILE): 1048576:1048576
notice getrlimit(RLIMIT_NOFILE): 1048576:1048576
notice start worker processes
notice start worker processes
notice start worker process 10588
notice start worker process 10588
notice signal 17 (SIGCHLD) received
notice signal 17 (SIGCHLD) received
info waitpid() failed (10: No child processes)
notice signal 17 (SIGCHLD) received
notice signal 17 (SIGCHLD) received
info waitpid() failed (10: No child processes)
notice [lua] start_sockproc.lua:11: auto-ssl: starting sockproc, context: init_worker_by_lua*
notice [lua] start_sockproc.lua:11: auto-ssl: starting sockproc, context: init_worker_by_lua*
CRIT reaped unknown pid 19899)
notice signal 17 (SIGCHLD) received
notice signal 17 (SIGCHLD) received
notice signal 17 (SIGCHLD) received
notice signal 17 (SIGCHLD) received
info waitpid() failed (10: No child processes)
```

I came across this investigation here (https://github.com/GUI/lua-resty-auto-ssl/issues/43#issuecomment-539993756) and decided to bump openresty to `openresty/openresty:1.15.8.2-5-xenial` to try and fix the lock issues in `lua-resty-core`.

The instances that we have running are more stable now, but we keep seeing this error when generating new certs:

```
error [lua] lets_encrypt.lua:40: issue_cert(): auto-ssl: dehydrated failed: env HOOK_SECRET=xxx HOOK_SERVER_PORT=8999 /usr/local/openresty/luajit/bin/resty-auto-ssl/dehydrated --cron --accept-terms --no-lock --domain xxxx.example.com --challenge http-01 --config /etc/resty-auto-ssl/letsencrypt/config --hook /usr/local/openresty/luajit/bin/resty-auto-ssl/letsencrypt_hooks status: 256 out: # INFO: Using main config file /etc/resty-auto-ssl/letsencrypt/config
ERROR # INFO: Using additional config file /etc/resty-auto-ssl/letsencrypt/conf.d/dehydrated-conf.sh
ERROR startup_hook
ERROR Processing xxxx.example.com
ERROR + Signing domains...
ERROR + Generating private key...
ERROR + Generating signing request...
ERROR + Requesting new certificate order from CA...
ERROR err: + ERROR: An error occurred while sending post-request to https://acme-v02.api.letsencrypt.org/acme/new-order (Status 400)
ERROR
ERROR Details:
ERROR HTTP/1.1 100 Continue
ERROR
ERROR HTTP/1.1 400 Bad Request
ERROR Server: nginx
ERROR Date: Wed, 27 Nov 2019 23:00:58 GMT
ERROR Content-Type: application/problem+json
ERROR Content-Length: 107
ERROR Connection: keep-alive
ERROR Boulder-Requester: 72675174
ERROR Cache-Control: public, max-age=0, no-cache
ERROR Link: ;rel="index"
ERROR Replay-Nonce: xxxx
ERROR
ERROR {
ERROR "type": "urn:ietf:params:acme:error:malformed",
ERROR "detail": "JWS verification error",
ERROR "status": 400
ERROR }
```

Have you seen this before? Is there a misconfiguration somewhere in our nginx config, or in dehydrated? I'm not sure if this is related to the locking issue or not, my guess would be no.

Thanks!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at lets_encrypt.lua:40 and inspect the dehydrated command, /etc/resty-auto-ssl/letsencrypt/config, and dehydrated-conf.sh. Reproduce certificate generation while reviewing the JWS verification error and determine whether the cause is in dehydrated, its configuration, or the OpenResty integration; done means a reproducible fix or a documented configuration correction.

Written by the indexing model from the issue text.

Assessment

Tech stack
lua, nginx, shell
Domain
backend, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.