Changing fig.set_size_inches() interactively shrinks the figure/canvas

Problem Description

I am trying to create a figure and subsequently change it using interactive input. Specifically, I would like to change the number of subplot axes based on the input. The following code works as expected:

%matplotlib widget

import matplotlib.pyplot as plt
import matplotlib.gridspec as gridspec
import ipywidgets as widgets
from IPython.display import display

# Create a plot with one (main) axes
fig, axs = plt.subplots(1, figsize = (10,5), sharex = True)
axs_height = 4
sub_axs_height = 0.2

number_widget = widgets.IntSlider(
    value = 1,
    min = 0,
    max = 5,
    step = 1,
    continuous_update = False,
    description = 'Sub-axes:'
    )

def editExistingFigure(number_of_additional_axes):
    # Clear the "main" axes
    axs.cla()
    # Remove any currently existing "additional" axes
    for ax in fig.axes[1:]: fig.delaxes(ax)
    # Set up the grid in preparation for the new axes
    gs = gridspec.GridSpec(
        nrows = 1 + number_of_additional_axes,
        ncols = 1,
        figure = fig,
        height_ratios = [axs_height] + [sub_axs_height]*number_of_additional_axes
        )
    # Place the "main" axes in the grid
    axs.set_position(gs[0,0].get_position(fig))
    # Plot something on the main axes
    axs.plot((1,2), (1,2))
    
    for i in range(number_of_additional_axes):
        # Add the additional axes
        additional_ax = fig.add_subplot(gs[i+1,0], sharex = axs)
        # Plot something on the new axes
        additional_ax.plot((1,2), (1,1))
        
ui = widgets.Box([number_widget])
out = widgets.interactive_output(
    editExistingFigure,
    {'number_of_additional_axes' : number_widget}
    )

# Display the UI with the output
display(ui, out)

The output looks like this, while the additional axes are added/removed as expected based on the interaction with the slider, and the size of the figure remains constant throughout:

However, I would like to keep the absolute size of the "main axes" constant, while changing the overall size of the figure based on the number of "additional axes." I was hoping I'd achieve that by removing the figsize setting from the initial plt.subplots() call and then setting fig.set_size_inches() based on the interactive input, as follows:

%matplotlib widget

import matplotlib.pyplot as plt
import matplotlib.gridspec as gridspec
import ipywidgets as widgets
from IPython.display import display

# Create a plot with one (main) axes
fig, axs = plt.subplots(1, sharex = True)    # figsize removed from here
axs_height = 4
sub_axs_height = 0.2

number_widget = widgets.IntSlider(
    value = 1,
    min = 0,
    max = 5,
    step = 1,
    continuous_update = False,
    description = 'Sub-axes:'
    )

def editExistingFigure(number_of_additional_axes):
    
    # Clear the main axes
    axs.cla()
    # Remove all currently existing "additional axes"
    for ax in fig.axes[1:]: fig.delaxes(ax)
    # Set up the grid in preparation for the new axes
    gs = gridspec.GridSpec(
        nrows = 1 + number_of_additional_axes,
        ncols = 1,
        figure = fig,
        height_ratios = [axs_height] + [sub_axs_height]*number_of_additional_axes
        )
    # Place the "main" axes in the grid
    axs.set_position(gs[0,0].get_position(fig))
    # Plot something on the main axes
    axs.plot((1,2), (1,2))
    
    for i in range(number_of_additional_axes):
        # Add the additional axes
        additional_ax = fig.add_subplot(gs[i+1,0], sharex = axs)
        # Plot something on the new axes
        additional_ax.plot((1,2), (1,1))
    
    # THIS LINE IS NEW:
    fig.set_size_inches(10, axs_height + sub_axs_height*number_of_additional_axes)
        
ui = widgets.Box([number_widget])
out = widgets.interactive_output(
        editExistingFigure,
        {'number_of_additional_axes' : number_widget}
        )

# Display the UI with the output
display(ui, out)

When the cell is run for the first time, the resulting figure looks identical to the one above. However, when the slider is changed, now the output canvas suddenly shrinks ~2-fold, as follows:

What is interesting is that the underlying figure seems to be the correct size but the canvas is what shrinks. Additionally, the size of the canvas is proportionate to the size of the underlying figure - i.e., it does respond to changes in the interactive input (height increases a little with slider number increasing, while the width is constant), but is just smaller.
The moment I interactively pull the right lower corner of the canvas with my mouse, the figure resizes to fit the canvas.

Note: This issue may be related to the following issues: #117, #36, and especially #17 (due to my impression that the output shrinks proportionately by ~2x and I am using a high-dpi diplay), though because there's no loop involved, it seems to be caused by fig.set_size_inches(), and none of the solutions suggested in the issues above worked, I decided to open a separate issue.

Versions

 3.7.3 (default, Mar 27 2019, 22:11:17) 
[GCC 7.3.0]
ipympl version: 0.5.6
jupyter core     : 4.5.0
jupyter-notebook : 5.7.9
qtconsole        : 4.5.1
ipython          : 7.6.0
ipykernel        : 5.1.1
jupyter client   : 5.2.4
jupyter lab      : 2.1.4
nbconvert        : 5.6.1
ipywidgets       : 7.5.1
nbformat         : 5.0.6
traitlets        : 4.3.2
Known nbextensions:
  config dir: /dartfs-hpc/rc/home/h/f002b9h/.conda/envs/scRNAseq/etc/jupyter/nbconfig
    notebook section
      jupyter-matplotlib/extension  enabled 
      - Validating: OK
      jupyter-js-widgets/extension  enabled 
      - Validating: OK
JupyterLab v2.1.4
Known labextensions:
   app dir: /dartfs-hpc/rc/home/h/f002b9h/.conda/envs/scRNAseq/share/jupyter/lab
        @jupyter-widgets/jupyterlab-manager v2.0.0  enabled  OK
        @lckr/jupyterlab_variableinspector v0.5.1  enabled  OK
        jupyter-matplotlib v0.7.2  enabled  OK

Start by running the notebook reproduction with %matplotlib widget, the slider, and fig.set_size_inches(), then compare it with the version that omits that call. Investigate the ipympl canvas resize behavior described in the issue; done means changing the slider preserves the intended canvas dimensions without requiring manual corner resizing.