Scroll after mount jumps all over the place

Hey,

I'm running into an issue where I'm fetching messages using the Tanstack Query Infinite Query function.
Works well but I get the most scroll behaviour imaginable:

It'll start at the bottom (as it should) but then as it fetches more data/refreshes the query cache it'll just jump to the start of the next page. I tried about anything to get rid of that behaviour but nothing seems to work.

Any idea what's up with that?