Inconsistent performance due to frequent unicorn worker restarts

I experienced some strange slow downs while navigating the gitlab web ui. While most of the requests are served in under 1 second, 20-30% of the requests needed 4-5 seconds to be served.

I digged a bit into it and found that the the unicorn worker restarts almost every 1-3 requests with just me navigating the web ui and 2 runners locking for jobs. I could even reproduced the problem on a fresh container with 9.5.3 by checking the unicorn.stderr.log while navigating the web ui.

Researching for a while gave me some more evidence that the frequent worker restarts might be the problem: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/2421

Unicorn workers while idle:

PID ELAPSED RSS COMMAND
583 02:36:41 327872 unicorn_rails master -c /home/git/gitlab/config/unicorn.rb -E production
671 02:36:19 385752 sidekiq 5.0.4 gitlab [0 of 25 busy]
6534 23:15 324424 unicorn_rails worker[1] -c /home/git/gitlab/config/unicorn.rb -E production
6548 23:03 324300 unicorn_rails worker[2] -c /home/git/gitlab/config/unicorn.rb -E production
6597 22:45 321052 unicorn_rails worker[0] -c /home/git/gitlab/config/unicorn.rb -E production

Unicorn workers after 3 web ui request (all restarted):

PID ELAPSED RSS COMMAND
583 02:38:46 328028 unicorn_rails master -c /home/git/gitlab/config/unicorn.rb -E production
671 02:38:24 385752 sidekiq 5.0.4 gitlab [0 of 25 busy]
7474 00:13 362032 unicorn_rails worker[1] -c /home/git/gitlab/config/unicorn.rb -E production
7488 00:08 352336 unicorn_rails worker[2] -c /home/git/gitlab/config/unicorn.rb -E production
7502 00:01 313600 unicorn_rails worker[0] -c /home/git/gitlab/config/unicorn.rb -E production

After some more research, i decided to increase the worker memory limit by building my own image from the source and adding the following lines to the unicorn.rb:

ENV['GITLAB_UNICORN_MEMORY_MIN'] = "629145600"
ENV['GITLAB_UNICORN_MEMORY_MAX'] = "734003200"

That results in a much smoother experience while navigating the web ui on an almost fresh 9.5.3 container. The workers no longer restarts all the time and the response times are much more stable.

Unicorn workers after launch:

PID ELAPSED RSS COMMAND
601 06:24 320224 unicorn_rails master -c /home/git/gitlab/config/unicorn.rb -E production
691 06:02 402704 sidekiq 5.0.4 gitlab [0 of 25 busy]
695 05:53 404116 unicorn_rails worker[0] -c /home/git/gitlab/config/unicorn.rb -E production
698 05:53 404528 unicorn_rails worker[1] -c /home/git/gitlab/config/unicorn.rb -E production
701 05:53 403504 unicorn_rails worker[2] -c /home/git/gitlab/config/unicorn.rb -E production

Unicorn workers after 50+ web ui requests:

PID ELAPSED RSS COMMAND
601 49:46 323320 unicorn_rails master -c /home/git/gitlab/config/unicorn.rb -E production
691 49:24 436396 sidekiq 5.0.4 gitlab [0 of 25 busy]
695 49:15 418644 unicorn_rails worker[0] -c /home/git/gitlab/config/unicorn.rb -E production
698 49:15 422732 unicorn_rails worker[1] -c /home/git/gitlab/config/unicorn.rb -E production
701 49:15 428004 unicorn_rails worker[2] -c /home/git/gitlab/config/unicorn.rb -E production

Later, i realised that i can change the worker memory limit by setting the GITLAB_UNICORN_MEMORY_MAX environment variable.

Finally, after many hours of research, i fixed the problem in our production environment by increasing the worker memory limit to 500 MB by adding the following environment variable to my docker-compose file:

GITLAB_UNICORN_MEMORY_MAX=524288000

It would be nice to have some additional documentation about these variables or even better, some more decent defaults, as the problem exists on a fresh container as well.

Start with config/unicorn.rb and the container environment handling for GITLAB_UNICORN_MEMORY_MIN and GITLAB_UNICORN_MEMORY_MAX; review unicorn.stderr.log while reproducing requests in a fresh container. Compare the default worker behavior with the reported 500 MB setting, then define whether the issue is resolved by documented configuration or by better defaults.