xarray 0.18.0 raises ValueError, not FileNotFoundError, when opening a non-existent file

What happened:

In a Python environment with xarray 0.18.0 and python-netcdf4 installed, I called xarray.open_dataset("nonexistent"). (The file "nonexistent" does not exist.) xarray threw a ValueError: cannot guess the engine, try passing one explicitly.

What you expected to happen:

I expected a FileNotFoundError error to be thrown, as in xarray 0.17.0.

Minimal Complete Verifiable Example:

import xarray as xr
xr.open_dataset("nonexistent")

Anything else we need to know?:

This is presumably related to Issue #5295, but is not fixed by PR #5296: ValueError is also thrown with the currently latest commit in master (9165c266).

This change in behaviour produced a hard-to-diagnose bug deep in xcube, where we were catching the FileNotFound exception to deal gracefully with a missing file, but the new ValueError was of course not caught -- and the error message did not make the cause obvious. Catching ValueError is a workaround, but not a great solution since it may also be thrown for files which do exist but don't have a recognizable data format. I suspect that other codebases may be similarly affected.

xarray 0.17.0 was capable of throwing a ValueError for a non-existent file, but only in the (rare?) case that neither netCDF4-python nor scipy was installed.

Environment:

Start by reproducing the reported xarray.open_dataset("nonexistent") behavior in xarray 0.18.0 and compare it with 0.17.0, then inspect the engine-selection path referenced by Issue #5295 and PR #5296. Done means a missing file raises FileNotFoundError while an existing file with an unrecognized format can still raise ValueError.