sameersbn / sameersbn/docker-gitlab

Unicorn does not come up (error 502) after hard restart of Docker server

Open
#1,305 27 comments 35 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Shell
Stars
8.1k
Forks
2.1k
Avg merge
17h 47m
Merged PRs (30d)
7

Description

Steps to reproduce
  1. Run GitLab using the guide
  2. Power cycle the server running Docker
Actual result

GitLab will never come up fully, showing error 502.

The docker container logs will have this:

2017-07-26 23:20:38,558 INFO spawned: 'unicorn' with pid 612
2017-07-26 23:20:39,160 INFO exited: unicorn (exit status 1; not expected)
...
2017-07-26 23:20:46,864 INFO spawned: 'unicorn' with pid 647
2017-07-26 23:20:47,312 INFO exited: unicorn (exit status 1; not expected)
2017-07-26 23:20:48,313 INFO gave up: unicorn entered FATAL state, too many start retries too quickly

unicorn_stderr.log will have this:

...
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:195:in `pid=': Already running on PID:601 (or pid=/home/git/gitlab/tmp/pids/unicorn.pid is stale) (ArgumentError)
        from /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:127:in `start'
        from /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/bin/unicorn_rails:209:in `<top (required)>'
        from /home/git/gitlab/vendor/bundle/ruby/2.3.0/bin/unicorn_rails:22:in `load'
        from /home/git/gitlab/vendor/bundle/ruby/2.3.0/bin/unicorn_rails:22:in `<main>'
Workaround

The only way to bring up GitLab will be to docker exec into the container, manually delete the stale pid file and restart the container:

docker exec -it gitlab rm /home/git/gitlab/tmp/pids/unicorn.pid && docker restart gitlab
Expected result

GitLab comes up without manual intervention.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Docker container startup behavior and the Unicorn PID path at /home/git/gitlab/tmp/pids/unicorn.pid; reproduce the hard-restart sequence and inspect unicorn_stderr.log. Trace how the container handles the stale PID after restart, and consider the issue done when GitLab starts without manually deleting the file.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, ruby
Domain
backend, devops, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.