Configurable max_http_connections of oc_bifrost (Was: oc_bifrost service malfunctioning on heavily loaded chef server (Ubuntu 14.04 LTS, chef 12.2.0-1))

(I opened this in the chef project first https://github.com/chef/chef/issues/4140, which wasn't the proper place... so I am opening this here again as @stevendanna has suggested...)

After we upgraded to chef server 12.2.0, it gave us lot of "internal server error 500"s on every 1st, 2nd or 3rd days.
It turned out that the cause of the problem was the oc_bifrost service.
After we had restarted the oc_bifrost service of chef, the phenomenon disappeared/problem was solved for the next 1, 2 or 3 days.

The probable cause could be the max connection limit(2048) of the mochiweb http module/processes of the oc_bifrost service.

The theory in short:
When the chef server gets heavy traffic and the number of active http connections reaches the max connection limit (2048) in oc_bifrost service (requests are coming from the opscode_erchef service) the newer http connections get into the backlog queue. Probably some of the opscode_erchef service connections (in the backlog) time out after 3 seconds(/etc/opscode/chef-server.rb: opscode_erchef['authz_timeout'] = 3000) which generates exceptions in the oc_bifrost service as the peer(opscode_erchef) terminates the connection... and the chef-clients get the 500 messages...

Maybe I am wrong (as I am not an erlang programmer) but I can imagine that the erlang supervisor doesn’t start the affected child processes after a couple of exceptions reached (restart policy: one_for_one/10/10)...

Unfortunately there is no way (or I didn’t find one) which we can use to configure the max http connections of the oc_bifrost service without modifying the erlang source code.

Relevant source codes:
In /opt/opscode/embedded/service/oc_bifrost/lib/mochiweb-2.9.0/src/mochiweb_socket_server.erl check "max=2048":

-record(mochiweb_socket_server,
        {port,
         loop,
         name=undefined,
         max=2048,
         ip=any,
         listen=null,
         nodelay=false,
         backlog=128,
         active_sockets=0,
         acceptor_pool_size=16,
         ssl=false,
         ssl_opts=[{ssl_imp, new}],
         acceptor_pool=sets:new(),
         profile_fun=undefined}).

In /opt/opscode/embedded/service/oc_bifrost/lib/bifrost-12.2.0/src/bifrost_sup.erl there is no param set for max:

    WebConfig = [
                 {ip, Ip},
                 {port, Port},
                 {log_dir, "priv/log"},
                 {dispatch, add_dynamic_config(Dispatch)},
                 {nodelay, true} % TCP 'no delay' for latency
                ],

So I took the bravery and I modified and compiled the relevant .erl file and restarted the oc_bifrost service.

root@chef:/opt/opscode/embedded/service/oc_bifrost/lib/bifrost-12.2.0/src# diff -u bifrost_sup.erl ~/bifrost_sup.erl.orig

--- bifrost_sup.erl     2015-10-08 11:59:56.077695357 +0000
+++ ~/bifrost_sup.erl.orig       2015-10-08 07:42:41.045186259 +0000
@@ -45,7 +45,6 @@
     WebConfig = [
                  {ip, Ip},
                  {port, Port},
-                 {max, 4096},
                  {log_dir, "priv/log"},
                  {dispatch, add_dynamic_config(Dispatch)},
                  {nodelay, true} % TCP 'no delay' for latency

Before my modification the oc_bifrost service died every 1st ,2nd or 3rd days. After my modification (which happened one month ago) the chef works fine without any glitch.

What do you think? Am I (or my conclusion) totally wrong? Is there any other known/existing solution to fix this oc_bifrost malfunctioning problem.

Can you please implement a chef server optional setting for tune the max_http_connections of the oc_bifrost service?

I mean for example in /etc/opscode/chef-server.rb:
oc_bifrost['max_http_connections'] = 4096

Start with bifrost_sup.erl and its WebConfig, then inspect mochiweb_socket_server.erl where the default max is defined. Trace how Chef Server settings in /etc/opscode/chef-server.rb reach oc_bifrost, and verify that an oc_bifrost['max_http_connections'] value changes the configured connection limit without editing Erlang source.