Matplotlib Fails to Update Axis Limits with ipywidgets in Jupyter Lab

Nobody has claimed this yet.

I'm using `ipywidgets` along with `matplotlib` in Jupyter Lab and stumbled across some strange behavior. Here's an example of the code:

```python
import matplotlib.pyplot as plt
import ipywidgets as widgets
%matplotlib widget
plt.ioff()

d = {
"a": [1,2,3],
"b": [2,3,4]
}

fig, ax = plt.subplots()

def update(change):
data = d[change["new"]]
ax.clear() # Clear the previous plot
ax.plot(data, data)
ax.set_xlim(min(data), max(data))
ax.set_ylim(min(data), max(data))
ax.figure.canvas.draw()

dropdown = widgets.Dropdown(options = list(d.keys()))
dropdown.observe(update, "value")

display(dropdown, fig.canvas)
update({"new": "a"})
```

The steps I carry out are as follows:

- Select `b` from dropdown
- Pan the plot
- Select `a` from dropdown
- Click on the "Reset original view" button on the plot

Instead of snapping the plot extents back to what they are set for choice `a`, I instead get the following:

[![enter image description here][1]][1]

Where it looks like `matplotlib` has indeed plotted the data `[1,2,3]`, but somehow failed to update the axis limits based on that data.

I really hope this isn't a bug, and that I'm doing something wrong. Has anyone seen this behavior before?

[1]: https://i.stack.imgur.com/nCOHG.png

Start by running the Jupyter Lab reproducer in the issue with the ipywidgets dropdown and Matplotlib widget backend, then follow the listed selection, pan, and reset sequence. Compare the restored axis limits with the limits set for each dataset; done means the reset view matches the currently selected data after switching from b to a.

Written by the indexing model from the issue text.