Engine parameter ignored when writing NetCDF4 if writing to a file handle.

What happened?

I tried to call ds.to_netcdf(f, engine="netcdf4", format="NETCDF4") and got an

ValueError: invalid format for scipy.io.netcdf backend: 'NETCDF4'

What did you expect to happen?

I expected the write to succeed.

Minimal Complete Verifiable Example

import numpy as np
import xarray as xr

arr_thr = np.random.random_sample((2, 100, 100))

x_var = np.arange(100)
y_var = np.arange(100)

ds = xr.Dataset(
    {
        "low_threshold": (["x", "y"], arr_thr[0], {"units": "dB"}),
        "high_threshold": (["x", "y"], arr_thr[1], {"units": "dB"}),
    },
    coords={
        "x_dist": (["x"], x_var, {"units": "m"}),
        "y_dist": (["y"], y_var, {"units": "m"}),
    },
)

with open("/tmp/test.nc", "wb") as f:
    ds.to_netcdf(f, engine="netcdf4", format="NETCDF4")

MVCE confirmation

• Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• Complete example — the example is self-contained, including all data and the text of any traceback.
• Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

Output exceeds the [size limit](command:workbench.action.openSettings?[). Open the full output data [in a text editor](command:workbench.action.openLargeOutput?1f881801-9039-4b0c-abe7-f26b42b290b9)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In [9], line 2
      1 with open("/tmp/test.nc", "wb") as f:
----> 2     ds.to_netcdf(f, engine="netcdf4", format="NETCDF4")

File ~/mambaforge/envs/eiland-dev/lib/python3.10/site-packages/xarray/core/dataset.py:1882, in Dataset.to_netcdf(self, path, mode, format, group, engine, encoding, unlimited_dims, compute, invalid_netcdf)
   1879     encoding = {}
   1880 from ..backends.api import to_netcdf
-> 1882 return to_netcdf(  # type: ignore  # mypy cannot resolve the overloads:(
   1883     self,
   1884     path,
   1885     mode=mode,
   1886     format=format,
   1887     group=group,
   1888     engine=engine,
   1889     encoding=encoding,
   1890     unlimited_dims=unlimited_dims,
   1891     compute=compute,
   1892     multifile=False,
   1893     invalid_netcdf=invalid_netcdf,
   1894 )

File ~/mambaforge/envs/eiland-dev/lib/python3.10/site-packages/xarray/backends/api.py:1193, in to_netcdf(dataset, path_or_file, mode, format, group, engine, encoding, unlimited_dims, compute, multifile, invalid_netcdf)
   1189     else:
...
--> 138     raise ValueError(f"invalid format for scipy.io.netcdf backend: {format!r}")
    140 if lock is None and mode != "r" and isinstance(filename_or_obj, str):
    141     lock = get_write_lock(filename_or_obj)

ValueError: invalid format for scipy.io.netcdf backend: 'NETCDF4'

Anything else we need to know?

No response

Environment

Start at Dataset.to_netcdf and the backend dispatch in xarray/backends/api.py, which appears in the traceback. Reproduce the file-handle example with engine="netcdf4" and format="NETCDF4". Done means the specified engine is honored and the example writes successfully without the scipy backend format error.