io_submit seems to be a blocking call

We run into issues because `io_submit` is slow. In fact it can be (in rare cases) so slow that a process gets dropped because it stops to heartbeat for too long.

@kaomakino created the small test program below that reproduces the issue. The results will depend on the speed of your disk (which in itself suggests that `io_submit` is not truly asynchronous).

The program does the following:
1. The main thread will write 4KB blocks into random locations within a 4GB file.
1. A second thread will cal `fdatasync` every 3 seconds (this roughly simulates commits in fdb).

Calls to `io_submit` will then take milliseconds to complete on a fast disk. On a very slow disk, one `io_submit` can take several seconds (we saw up to 13 seconds).

This can be made better by doing the following:
1. Create a 4GiB file (for example with `dd` - content doesn't matter)
1. Change the attached C program so that it doesn't create the file when it opens it.

However, even than `io_submit` can take a very long time (~1ms on a fast disk). So this seems to be only part of the solution. Additionally, it seems to be the case that latencies of `io_submit` get better the smaller we make the write queue. With only 1 outstanding request at a time it seems to be 10-60 microseconds.

[aiotime.c.zip](https://github.com/apple/foundationdb/files/3661063/aiotime.c.zip)

Start with the attached aiotime.c reproducer and run it while varying disk speed, file creation, and the number of outstanding writes. Read the io_submit and fdatasync call paths involved in the test and compare the observed latency. Done means identifying the cause of the blocking behavior and documenting or implementing a verified resolution.

Written by the indexing model from the issue text.