Downsample with RangeTool source sometimes ignores set height

#### ALL software version info
(this library, plus any other relevant software, e.g. bokeh, python, notebook, OS, browser, etc)

#### Description of expected behavior and the observed behavior
**Sometimes** when **just** the source plot of a RangeToolLink is `downsample=True`, the set height for the source plot is ignored.

In the video below, after setting the target plot `downsample=False`, sometimes when the cell is executed, the height for the source plot is changes.

#### Complete, minimal, self-contained example code that reproduces the issue

```
import pandas as pd
import numpy as np
import hvplot.pandas
from holoviews.plotting.links import RangeToolLink

n_samples = 1000
data = np.random.randn(n_samples)

df = pd.DataFrame({'time': np.arange(len(data)), 'value': data})
plot = df.hvplot(x="time", y="value", height=400, downsample=False)
minimap = df.hvplot(x="time", y="value", height=150, downsample=True)
link = RangeToolLink(minimap, plot, axes=["x", "y"])

(plot + minimap).opts(shared_axes=False).cols(1)

```

#### Stack traceback and/or browser JavaScript console output

#### Screenshots or screencasts of the bug in action

https://github.com/holoviz/hvplot/assets/6613202/bf71119b-bd5a-4b91-a406-c9e92b9d633b

- [ ] I may be interested in making a pull request to address this