auto-ssl / auto-ssl/lua-resty-auto-ssl

Openresty process becoming unresponsive. Possibly lock related?

Open
#136 30 comments 0 reactions 0 assignees View on GitHub
Dominant language
Lua
Stars
2k
Forks
184
PR merge metrics
No merged PRs in 30d

Description

Hey! Sorry to bother you again, I think the instability that I had the other day (#135) was a red herring and not actually the cause of the outages that we've been having.

I've been seeing some things in our logs that look like the following:

```
Restarting selected processes ['openresty']
```

This is coming from [supervisor](http://supervisord.org/) detecting that openresty has become unresponsive. Scrolling to those logs, i see lots of things regarding a "shared memory zone" (i've redacted some duplicates for brevity):

```
[notice] 24883#24883: signal 15 (SIGTERM) received, exiting
[notice] 24883#24883: signal 14 (SIGALRM) received
[notice] 24883#24883: signal 17 (SIGCHLD) received
[alert] 24883#24883: worker process 24895 exited on signal 9
[alert] 24883#24883: shared memory zone "auto_ssl" was locked by 24895
[notice] 24883#24883: exit
[notice] 31640#31640: using the "epoll" event method
[notice] 31640#31640: openresty/1.11.2.2
[notice] 31640#31640: built by gcc 4.9.2 (Debian 4.9.2-10)
[notice] 31640#31640: OS: Linux 4.4.0-96-generic
[notice] 31640#31640: getrlimit(RLIMIT_NOFILE): 1048576:1048576
[notice] 31640#31640: start worker processes
[notice] 31640#31640: start worker process 31652
[notice] 31652#31652: signal 17 (SIGCHLD) received
[notice] 31652#31652: unknown process 31653 exited with code 0
[notice] 31652#31652: signal 17 (SIGCHLD) received
[notice] 31652#31652: unknown process 31655 exited with code 0
[notice] 31652#31652: *1 [lua] start_sockproc.lua:10: auto-ssl: starting sockproc, context: init_worker_by_lua*
[notice] 31652#31652: signal 17 (SIGCHLD) received
[info] 31652#31652: waitpid() failed (10: No child processes)
```

This log output lead me to the following issues:
https://github.com/GUI/lua-resty-auto-ssl/issues/43
https://github.com/18F/api.data.gov/issues/325

And a commit which fixed the locking issue: https://github.com/GUI/lua-resty-auto-ssl/commit/8bcb9ee83c1faae849ad456503f841afb804ad75

One of the error messages in that commit "failed to unlock:" is also coming up in our logs with the following error message:

```
[error] 3706#3706: *20191 [lua] ssl_certificate.lua:53: issue_cert_unlock(): auto-ssl: failed to unlock: lock does not match expected value, context: ssl_certificate_by_lua*
```

Which appears to come from: https://github.com/GUI/lua-resty-auto-ssl/blob/0fe6ab67f091d75821a865d8d71122dc04e8ff0d/lib/resty/auto-ssl/ssl_certificate.lua#L53

Have you ever seen this before? Is there anything I can do to debug this further? I've already bumped the auto_ssl `lua_shared_dict` to 100mb in an attempt to mitigate in the chance we were filling it up.

Tracing through the code to find places that may be calling this, I can see that I have some of these errors too: https://github.com/GUI/lua-resty-auto-ssl/blob/0fe6ab67f091d75821a865d8d71122dc04e8ff0d/lib/resty/auto-ssl/ssl_certificate.lua#L77

With `timeout` being the reason the lock couldnt be created:

```
[error] 1850#1850: *576 [lua] ssl_certificate.lua:68: issue_cert(): auto-ssl: failed to obtain lock: timeout, context: ssl_certificate_by_lua*
```

We're using redis as our storage adapter, if that matters? Thanks again for all of your help with this. If there's anything I can do to contribute back to the code for all that you've done, i'd be more than happy to.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with lib/resty/auto-ssl/ssl_certificate.lua at the referenced issue_cert_unlock() and issue_cert() locations, then review the linked locking issues and commit 8bcb9ee. Compare the reported shared-memory, failed-unlock, and lock-timeout logs with the Redis storage adapter behavior. Done requires isolating the cause and defining a confirmed fix or reproducible diagnostic.

Written by the indexing model from the issue text.

Assessment

Tech stack
lua, nginx, redis
Domain
security
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.