Listview: can get out of sync using screen reader with async lists

Firstly I'm not sure how this could possibly be fixed.

When using screen readers in a virtual buffer mode and navigating around tableview or listview it is possible to get into a situation where up and down navigate through lists in the wrong direction.

To reproduce go to https://react-spectrum.adobe.com/react-spectrum/ListView.html#asynchronous-loading

Using NVDA + Chrome (although I think this reproduces similarly in other screen readers that have a virtual cursor mode)
Press T to go the table
use Ctrl + Alt + up/down arrow (or ctrl+option and up/down when using a mac keyboard) to navigate through the table. Note that sometimes you skip multiple rows after you hit the edge of the loaded rows and then sometimes in these situations when you move in the opposite direction the row number moves in the wrong direction.

I have a video showing this but can't upload here by the looks of it.