read_callback called infrequently with PulseAudio

I'm building an application that uses audio interactively, meaning that I need to get audio from the device with relatively low latency (<20ms). However when using PulseAudio I get large chunks of data every ~300ms, even when I set software_latency to significantly less than that.

I used the [this test program](https://gist.github.com/jacquesh/2bfa38e73912a1b96271808a728d175a) on Ubuntu 16.04 with PulseAudio 8.0, statically linking to libsoundio 1.1.0 (compiled from source) and here is a sample of the output I got:
```
0.309 - Tick!
0.318 - Read callback! 0 - 8056 (1 channel)
0.318 - Read callback! 0 - 8056 (1 channel)
0.318 - Read callback! 0 - 6952 (1 channel)
0.318 - Read callback! 0 - 6952 (1 channel)
0.409 - Tick!
0.509 - Tick!
0.609 - Tick!
0.639 - Read callback! 0 - 1376 (1 channel)
```
You can see the [full output here](https://gist.github.com/jacquesh/6aade8c7d19ab5a183812eac35f15e00), the numbers down the left represent time (in seconds).
I get the same effect using the latest commit to master (at the time of writing fc96baf).
One additional strange aspect of this is that if you look at the number of frames provided to each invocation of the read_callback, you see that each value occurs twice and that it adds up to a total of ~600ms of audio every 300ms.

In the example program I use the device's software_latency_min as the latency when opening the input stream. This is 0.00 for my test case. If I instead specify 0.01 then I get the same effect but the read callback gets called far more frequently ([here's an example](https://gist.github.com/jacquesh/72248737e6fb246d9f9b06b2cbd81e44)).

If I run the same tests on Windows (using WASAPI) [this output](https://gist.github.com/jacquesh/0bc9cf078e59e5db6212e0c868301f37) when using software_latency_min (which, again, is 0.00) and [this output](https://gist.github.com/jacquesh/c27eb4eead4321d7dee9d063a78e29c8) when using 0.01. As you can see when I ask for a latency of 0.01 I get updates at consistent intervals, which is the desired behaviour.

This actually seems like two different issues to me, theres the phenomenon of the latency being high when you ask for a latency of 0 (which I do not see anything about in the docs so I would expect that it isn't the desired behaviour) and then the issue of PulseAudio giving data back in bursts.