provide "slow ramp time" to protect new app instances from overloading

Proposed Change

As a developer I would like to have a "slow ramp time" for app instances that have just become healthy. The behaviour is 100% similar to the F5 slow ramp time.
The gorouter should know the uptime of individual app instances, and for instances that just became healthy, it should not send the full share of load to it immediately, but rather slowly increase the request rate over a period specified by the slow_ramp_time.

This should provide better survivability for new app instances that need to warm up first (jit compile code, warm up backend connection pools etc..) before they provide good enough response times.

Acceptance criteria

A common scenario these days is the following:

We have a Java app running 10 instances by default. The app uses the App Autoscaler to increase the number of instances during high load periods.
However, when additional app instances have become healthy, they immediately get their share of the load which they cannot or hardly handle because Java code needs to be jitted first, connection pools initialized and so on, this results in excessive response times and outages for some customers, or the instance becomes completely unresponsive and CF kills it because the health-check times out.
As a result some teams decide to not use this dynamic scaling but just deploy the maximum number of instances all the time, which adds up to the costs.

A similar scenario is when an instance crashes (for whatever reason), it gets automatically restarted but keeps on crashing once the full load comes in again. Only drastically increasing the health-check timeout can help (but that requires a full redeployment of the app).

If the gorouter would be able to gradually increase the load to new instances over the given ramp time (~ 10-30 secs), then we expect these new instances to survive and provide better average response times and/or fewer slow responses.

It could be implemented as a regular gorouter (boshrelease) configuration, or (even better) as a per-route option.

Related links

No response

Start by locating the gorouter code that tracks healthy app instances and distributes requests; the issue names no file or test. Review whether the setting belongs in the BOSH release configuration or as a per-route option, with done defined as gradually increasing traffic to newly healthy instances over the requested ramp period.