bug(COMPONENT): CDK Virtual Scroller jump back/flickers to items on top

### Is this a regression?

- [ ] Yes, this behavior used to work in the previous version

### The previous version in which this bug was not present was

_No response_

### Description

I'm getting an annoying behaviour while scrolling, where items snap/jump back and I'm never able to scroll all the way down the end of the list. I've notice though, that this bug happens only when I use the virtual scrolling within a material dialog

Template

```

{{scan.value.name}}

```

CSS

`
.scroll-viewport {
height: 200px;
}
`

### Reproduction

- Implement **cdk-virtual-scroll-viewport** scroller **inside a Angular Material dialog**
- in the template set **templateCacheSize:0**
- try to scroll down

### Expected Behavior

https://github.com/angular/components/assets/6199235/63111f97-c665-4cc0-8f8b-eead8d75a025

### Actual Behavior

While scrolling down, the scroll jump up to the first times

### Environment

- Angular: ~11.2.14
- CDK/Material: ~11.2.13
- Browser(s): Chrome
- Operating System (e.g. Windows, macOS, Ubuntu): macOs Catalina

Reproduce the issue with cdk-virtual-scroll-viewport inside an Angular Material dialog, using templateCacheSize:0 and the provided 200px viewport. Trace the virtual scrolling and dialog behavior while scrolling downward. Done means the list can be scrolled to the end without jumping back to the first items or flickering.

Written by the indexing model from the issue text.