FDB storage process hangs if IO operation doesn't complete

So we experienced this quite interesting failure scenario on a production cluster:

Due to a strange disk failure, an IO-operation did not return (got blocked forever). Usually disk failures result in errors, but not in this case. This local problem grew into a global one:

1. The storage server in question didn't die. Instead the `updateStorage` actor (think) was waiting forever on a disk operation.
2. Due to that, the storageserver did not fetch or pop data from the TLogs
3. As the storage role didn't die, the data distribution layer didn't move the partitions away.
4. The TLogs started to grow unbounded and eventually created an alert

Now there is not a lot we can do against disk failures in general. However, a failing disk should never be able to bring down a cluster.

So I would propose the following change: If a filesystem operation takes longer than X seconds (X would need to be determined - I think something like 300 might be a good default) to complete, we should throw an error and kill the corresponding role on that process (without restarting it). That way a disk failure would simply remove that process and the data distributor could do its thing.

Any thoughts on that?

Start by tracing the updateStorage actor and the filesystem operations used by the storage server, then examine how TLogs, storage roles, and the data distributor interact when a role stops responding. Determine where an operation timeout and role termination would belong, including the timeout value. Done means a blocked filesystem operation causes the corresponding role to exit without restarting and prevents unbounded TLog growth.

Written by the indexing model from the issue text.