Nuxt SSR is not rendering correctly

I've a client-side Vue app, which I am migrating to render server-side with Nuxt. The client app is running fine, and SSR version has the same code without any changes.
SSR version looks good on the first render, but as soon as I scroll it very little (15-20 pixels), it starts hiding the rows by giving them -1000px translation. Basically, the lib is still doing the same that it should but way before the view is out of the parent div. I've 7-8 visible rows on desktop and they all disappear within 100px of scrolling.
Also when I scroll back, the rows are miss-aligned..placed on top of one another randomly.
I suspect there is some browser-based initialization is going under the hood which is causing the issue since it can't be calculated correctly and hence the strange behavior.

FYI: I've tried putting the Recycle Scroller in `` tags, as well as conditionally disable their parent from rendering based on `process.client` but still the same result.