WORKER_TTL is overloaded

The documentation for the WORKER_TTL option says that it specifies how long we'll wait for heartbeats before deciding that a worker is dead, which is true.

However it also is used as the "graceful shutdown wait period" if the worker process receives a SIGTERM or SIGINT. It's used this way in a somewhat roundabout way, heartbeat_period is set to WORKER_TTL / 3, and TASK_GRACE_INTERVAL is set to an unconfigurable 3.

I have an interest in increasing the graceful shutdown wait period, which I have done in my own deployment simply by increasing WORKER_TTL because there seemed to be very little downside to the other side-effects of that option (time to declare dead increase, heartbeat_interval increase). However there's very little reason that I can see that the graceful shutdown period has to equal the time-to-declare-dead period.

I would say at the very least the documentation on WORKER_TTL should be updated to indicate that it's also the graceful shutdown wait period, and that if you're using Kubernetes or Docker or whatever to run your workers then you should align this value with the configured or default values of terminationGracePeriodSeconds or timeout/stop_grace_period respectively. Or, possibly, I can see a case being made that this should be a separate config value, with appropriate docs.

I am not highly opinionated about the change that should be made here since I have already solved my own problem, I open this as a sparking point for discussion.

Start with the WORKER_TTL documentation at the linked worker-ttl settings page, then trace the mentioned heartbeat_period and TASK_GRACE_INTERVAL behavior. Decide whether the work is documentation-only or requires a separate graceful-shutdown setting, and define how Kubernetes or Docker grace periods should be aligned before making a change.