The position is reset when the scroll position of the listview is cached

I'm caching the ListView for page navigation(using a PageStorageKey). So that users won't have to scroll from the top each time. But I noticed the draggable scrollbar is reset when the page is navigated back. So I cannot drag the handler to the top (which is already in the top) when the list view is still in the middle.
I think it should keep sync to the list view.