auto-ssl / auto-ssl/lua-resty-auto-ssl
500 Internal Server Error when deploy_cert
- Dominant language
- Lua
- Stars
- 2k
- Forks
- 184
- PR merge metrics
- No merged PRs in 30d
Description
Hi! We're using auto-ssl + openresty to generate several certificates every day, but for some reason sometimes we're facing this issue when trying to generate a new certificate (we haven't still identified a pattern).
This is the full error we're getting when it's failing to generate a new SSL certificate:
```
2020/06/30 07:51:32 [error] 22#22: *18639769 lua entry thread aborted: runtime error: ...sty/luajit/share/lua/5.1/resty/auto-ssl/servers/hook.lua:44: assertion failed!
stack traceback:
coroutine 0:
[C]: in function 'assert'
...sty/luajit/share/lua/5.1/resty/auto-ssl/servers/hook.lua:44: in function 'server'
.../local/openresty/luajit/share/lua/5.1/resty/auto-ssl.lua:95: in function 'hook_server'
content_by_lua(nginx.conf:365):2: in main chunk, client: 127.0.0.1, server: , request: "POST /deploy-cert HTTP/1.1", host: "127.0.0.1:8999"
2020/06/30 07:51:32 [error] 22#22: *18639705 [lua] lets_encrypt.lua:77: issue_cert(): auto-ssl: dehydrated manual hook.sh failed: env HOOK_SECRET=XXXXXXXXX HOOK_SERVER_PORT=8999 /usr/local/openresty/luajit/bin/resty-auto-ssl/letsencrypt_hooks deploy_cert myawesomedomain.com /srv/domains/letsencrypt/letsencrypt/certs/myawesomedomain.com/privkey.pem /srv/domains/letsencrypt/letsencrypt/certs/myawesomedomain.com/cert.pem /srv/domains/letsencrypt/letsencrypt/certs/myawesomedomain.com/fullchain.pem /srv/domains/letsencrypt/letsencrypt/certs/myawesomedomain.com/chain.pem 1593503492 status: 256 out: deploy_cert
err: curl: (22) The requested URL returned error: 500 Internal Server Error
hook request (deploy_cert) failed
, context: ssl_certificate_by_lua*, client: 95.169.228.179, server: 0.0.0.0:443
2020/06/30 07:51:32 [error] 22#22: *18639705 [lua] ssl_certificate.lua:97: issue_cert(): auto-ssl: issuing new certificate failed: dehydrated failure, context: ssl_certificate_by_lua*, client: 95.169.228.179, server: 0.0.0.0:443
2020/06/30 07:51:32 [error] 22#22: *18639705 [lua] ssl_certificate.lua:291: auto-ssl: could not get certificate for myawesomedomain.com - using fallback - failed to get or issue certificate, context: ssl_certificate_by_lua*, client: 95.169.228.179, server: 0.0.0.0:443
```
In our case, we're just storing the SSL certificates in a NFS volume, where we've 2 servers behind a load balancer processing the requests and mounting the NFS volume. Checking the directory **/srv/domains/letsencrypt/letsencrypt/certs/myawesomedomain.com/** looks like it has already generated some certificates:
```
root@localhost:/srv/domains/letsencrypt/letsencrypt/certs/myawesomedomain.com# ls -lrt
total 276
-rw------- 1 www-data www-data 3243 May 25 20:41 privkey-1590439289.pem
-rw------- 1 www-data www-data 1675 May 25 20:41 cert-1590439289.csr
-rw------- 1 www-data www-data 0 May 25 20:41 cert-1590439289.pem
-rw------- 1 www-data www-data 3247 May 25 20:41 privkey-1590439305.pem
-rw------- 1 www-data www-data 1675 May 25 20:41 cert-1590439305.csr
-rw------- 1 www-data www-data 0 May 25 20:41 cert-1590439305.pem
-rw------- 1 www-data www-data 3243 May 25 20:42 privkey-1590439320.pem
-rw------- 1 www-data www-data 1675 May 25 20:42 cert-1590439320.csr
-rw------- 1 www-data www-data 0 May 25 20:42 cert-1590439320.pem
-rw------- 1 www-data www-data 3243 May 25 20:42 privkey-1590439325.pem
-rw------- 1 www-data www-data 1675 May 25 20:42 cert-1590439325.csr
-rw------- 1 www-data www-data 0 May 25 20:42 cert-1590439325.pem
-rw------- 1 www-data www-data 3243 May 25 20:42 privkey-1590439335.pem
-rw------- 1 www-data www-data 1675 May 25 20:42 cert-1590439335.csr
-rw------- 1 www-data www-data 0 May 25 20:42 cert-1590439335.pem
-rw------- 1 www-data www-data 3247 May 25 20:42 privkey-1590439349.pem
-rw------- 1 www-data www-data 1675 May 25 20:42 cert-1590439349.csr
-rw------- 1 www-data www-data 0 May 25 20:42 cert-1590439349.pem
-rw------- 1 www-data www-data 3243 May 25 20:42 privkey-1590439350.pem
-rw------- 1 www-data www-data 1675 May 25 20:42 cert-1590439350.csr
-rw------- 1 www-data www-data 0 May 25 20:42 cert-1590439350.pem
-rw------- 1 www-data www-data 3247 May 25 20:42 privkey-1590439355.pem
-rw------- 1 www-data www-data 1675 May 25 20:42 cert-1590439355.csr
-rw------- 1 www-data www-data 0 May 25 20:42 cert-1590439355.pem
-rw------- 1 www-data www-data 3247 May 25 20:42 privkey-1590439364.pem
-rw------- 1 www-data www-data 1675 May 25 20:42 cert-1590439364.csr
-rw------- 1 www-data www-data 0 May 25 20:42 cert-1590439364.pem
-rw------- 1 www-data www-data 3243 May 25 20:42 privkey-1590439365.pem
-rw------- 1 www-data www-data 1675 May 25 20:42 cert-1590439365.csr
-rw------- 1 www-data www-data 0 May 25 20:42 cert-1590439365.pem
-rw------- 1 www-data www-data 3243 May 25 20:42 privkey-1590439368.pem
-rw------- 1 www-data www-data 1675 May 25 20:42 cert-1590439368.csr
-rw------- 1 www-data www-data 0 May 25 20:42 cert-1590439368.pem
-rw------- 1 www-data www-data 3243 May 25 20:42 privkey-1590439370.pem
-rw------- 1 www-data www-data 1675 May 25 20:42 cert-1590439370.csr
-rw------- 1 www-data www-data 0 May 25 20:42 cert-1590439370.pem
-rw------- 1 www-data www-data 3247 May 25 20:42 privkey-1590439372.pem
-rw------- 1 www-data www-data 1675 May 25 20:42 cert-1590439372.csr
-rw------- 1 www-data www-data 0 May 25 20:42 cert-1590439372.pem
-rw------- 1 www-data www-data 3243 May 25 20:42 privkey-1590439374.pem
-rw------- 1 www-data www-data 1675 May 25 20:42 cert-1590439374.csr
-rw------- 1 www-data www-data 0 May 25 20:42 cert-1590439374.pem
-rw------- 1 www-data www-data 3243 May 25 20:42 privkey-1590439377.pem
-rw------- 1 www-data www-data 1675 May 25 20:42 cert-1590439377.csr
-rw------- 1 www-data www-data 0 May 25 20:42 cert-1590439377.pem
-rw------- 1 www-data www-data 3243 May 25 20:42 privkey-1590439379.pem
-rw------- 1 www-data www-data 1675 May 25 20:42 cert-1590439379.csr
-rw------- 1 www-data www-data 0 May 25 20:42 cert-1590439379.pem
-rw------- 1 www-data www-data 3243 May 25 21:36 privkey-1590442609.pem
-rw------- 1 www-data www-data 1675 May 25 21:36 cert-1590442609.csr
-rw------- 1 www-data www-data 0 May 25 21:36 cert-1590442609.pem
-rw------- 1 www-data www-data 3243 May 25 21:36 privkey-1590442612.pem
-rw------- 1 www-data www-data 1675 May 25 21:36 cert-1590442612.csr
-rw------- 1 www-data www-data 0 May 25 21:36 cert-1590442612.pem
-rw------- 1 www-data www-data 3243 May 25 21:36 privkey-1590442614.pem
-rw------- 1 www-data www-data 1675 May 25 21:36 cert-1590442614.csr
-rw------- 1 www-data www-data 0 May 25 21:36 cert-1590442614.pem
-rw------- 1 www-data www-data 3243 May 25 21:36 privkey-1590442618.pem
-rw------- 1 www-data www-data 1675 May 25 21:36 cert-1590442618.csr
-rw------- 1 www-data www-data 0 May 25 21:36 cert-1590442618.pem
-rw------- 1 www-data www-data 3243 May 25 21:42 privkey-1590442973.pem
-rw------- 1 www-data www-data 1675 May 25 21:42 cert-1590442973.csr
-rw------- 1 www-data www-data 2277 May 25 21:43 cert-1590442973.pem
-rw------- 1 www-data www-data 1648 May 25 21:43 chain-1590442973.pem
lrwxrwxrwx 1 www-data www-data 20 May 25 21:43 chain.pem -> chain-1590442973.pem
lrwxrwxrwx 1 www-data www-data 24 May 25 21:43 fullchain.pem -> fullchain-1590442973.pem
lrwxrwxrwx 1 www-data www-data 19 May 25 21:43 cert.csr -> cert-1590442973.csr
lrwxrwxrwx 1 www-data www-data 19 May 25 21:43 cert.pem -> cert-1590442973.pem
-rw------- 1 www-data www-data 3925 May 25 21:43 fullchain-1590442973.pem
```
And the only way to recover from this error is just deleting the directory **/srv/domains/letsencrypt/letsencrypt/certs/myawesomedomain.com** and then forcing new request on the domain looks like the certificate is generated correctly. This is the version we're using at the moment:
```
lua-resty-auto-ssl
0.13.1-1 (installed) - /usr/local/openresty/luajit/lib/luarocks/rocks-5.1
```
Any idea of what might cause this problem?
Contributor guide
No contributing guide indexed for this repository
Research direction
Trace the assertion at servers/hook.lua:44 through auto-ssl.lua:95 and the letsencrypt_hooks deploy_cert entry point, then compare it with the issue_cert and ssl_certificate.lua callers. Reproduce certificate issuance with two OpenResty/nginx servers sharing the NFS certificate directory, and consider the issue done when the failure cause is identified and issuance no longer leaves unusable certificate files.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- nginx
- Domain
- infrastructure, security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100