auto-ssl / auto-ssl/lua-resty-auto-ssl

Locking problem

Open
#43 20 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Lua
Stars
2k
Forks
184
PR merge metrics
No merged PRs in 30d

Description

I've been encountering a problem with our (reverse proxy) nginx servers that they have been crashing. They stop responding to requests completely, and don't seem to come out of this state by themselves. These servers deal with a highish volume of requests (>5million a day).

For the past few days I've been at a bit of a loss, and restarting the Docker instance manually whenever I was alerted to this by monitoring, but I decided to put a helper cron script in place that would check if nginx was still responding and restart it via supervisord if there was an issue.

Due to the fact initially I was restarting the Docker container, I wasn't really getting any form of debugging information -- the logging would just stop. However after changing this to instead restart nginx inside the container I have the following in the logs:

```
2017/02/01 01:14:16 [alert] 489#0: worker process 501 exited on signal 9
2017/02/01 01:14:16 [alert] 489#0: shared memory zone "auto_ssl" was locked by 501
2017/02/01 01:14:16 [alert] 489#0: worker process 502 exited on signal 9
```

I had around on Google and the only reference I can find is https://github.com/18F/api.data.gov/issues/325 -- however it looks like expirations were put into place, this doesn't seem to be working on our setup, as we (due to bad monitoring) ended up with about a 7 hour downtime recently.

I should mention I cannot recreate this bug at all locally, even using the same Docker container.

I'm at a bit of a loss, our automatic restart script has sorted out the issues for now but it would be nice to see if anyone has ideas. I'd be happy to turn on extra logging and attempting the debug log (I've been a bit scared to turn it on in our production servers).

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the nginx error-log entries for the auto_ssl shared-memory lock and the production Docker/supervisord restart path; the issue names no repository file or test. Compare the high-volume production setup with the non-reproducing local container and collect the extra or debug logging offered in the report. Done means identifying a reproducible cause and documenting a verified fix or diagnostic next step.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, lua, nginx
Domain
backend, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.