fsspec / fsspec/filesystem_spec
HTTP client_kwargs authentication not used with simplecache
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 490
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 38
Description
Using fsspec='0.8.4+70.g33d532f' if a simplecache:: prefix is used with HTTP:// it seems that authentication settings are not used leading to ClientResponseError: 401, message='Unauthorized'
This works:
import fsspec
import xarray as xr
import netrc
import aiohttp
(username, account, password) = netrc.netrc().authenticators("urs.earthdata.nasa.gov")
uri = 'https://grfn.asf.alaska.edu/door/download/S1-GUNW-A-R-137-tops-20200311_20200305-020013-43055N_41518N-PP-00fe-v2_0_2.nc'
with fsspec.open(item.data.urlpath,
client_kwargs={'auth': aiohttp.BasicAuth(username, password)}) as f:
ds = xr.open_dataset(f, group='/science/grids/data', engine='h5netcdf', chunks={})
This does not:
with fsspec.open('simplecache::'+item.data.urlpath,
client_kwargs={'auth': aiohttp.BasicAuth(username, password)}) as f:
ds = xr.open_dataset(f, group='/science/grids/data', engine='h5netcdf', chunks={})
---------------------------------------------------------------------------
ClientResponseError Traceback (most recent call last)
<ipython-input-17-a30d2d9da865> in <module>
1 # with simplecache
2 with fsspec.open('simplecache::'+item.data.urlpath,
----> 3 client_kwargs={'auth': aiohttp.BasicAuth(username, password)}) as f:
4 ds = xr.open_dataset(f, group='/science/grids/data', engine='h5netcdf', chunks={})
5
~/GitHub/filesystem_spec/fsspec/core.py in __enter__(self)
100 mode = self.mode.replace("t", "").replace("b", "") + "b"
101
--> 102 f = self.fs.open(self.path, mode=mode)
103
104 self.fobjects = [f]
~/GitHub/filesystem_spec/fsspec/implementations/cached.py in <lambda>(*args, **kw)
384 # all the methods defined in this class. Note `open` here, since
385 # it calls `_open`, but is actually in superclass
--> 386 return lambda *args, **kw: getattr(type(self), item)(self, *args, **kw)
387 if item in ["__reduce_ex__"]:
388 raise AttributeError
~/GitHub/filesystem_spec/fsspec/spec.py in open(self, path, mode, block_size, cache_options, **kwargs)
912 autocommit=ac,
913 cache_options=cache_options,
--> 914 **kwargs
915 )
916 if not ac:
~/GitHub/filesystem_spec/fsspec/implementations/cached.py in <lambda>(*args, **kw)
384 # all the methods defined in this class. Note `open` here, since
385 # it calls `_open`, but is actually in superclass
--> 386 return lambda *args, **kw: getattr(type(self), item)(self, *args, **kw)
387 if item in ["__reduce_ex__"]:
388 raise AttributeError
~/GitHub/filesystem_spec/fsspec/implementations/cached.py in _open(self, path, mode, **kwargs)
637 f2.write(data)
638 else:
--> 639 self.fs.get(path, fn)
640 return self._open(path, mode)
641
~/GitHub/filesystem_spec/fsspec/asyn.py in get(self, rpath, lpath, recursive, **kwargs)
279 lpaths = other_paths(rpaths, lpath)
280 [os.makedirs(os.path.dirname(lp), exist_ok=True) for lp in lpaths]
--> 281 return sync(self.loop, self._get, rpaths, lpaths)
282
283
~/GitHub/filesystem_spec/fsspec/asyn.py in sync(loop, func, callback_timeout, *args, **kwargs)
69 if error[0]:
70 typ, exc, tb = error[0]
---> 71 raise exc.with_traceback(tb)
72 else:
73 return result[0]
~/GitHub/filesystem_spec/fsspec/asyn.py in f()
53 if callback_timeout is not None:
54 future = asyncio.wait_for(future, callback_timeout)
---> 55 result[0] = await future
56 except Exception:
57 error[0] = sys.exc_info()
~/GitHub/filesystem_spec/fsspec/asyn.py in _get(self, rpaths, lpaths, **kwargs)
267 *[
268 self._get_file(rpath, lpath, **kwargs)
--> 269 for lpath, rpath in zip(lpaths, rpaths)
270 ]
271 )
~/GitHub/filesystem_spec/fsspec/implementations/http.py in _get_file(self, rpath, lpath, chunk_size, **kwargs)
177 if r.status == 404:
178 raise FileNotFoundError(rpath)
--> 179 r.raise_for_status()
180 with open(lpath, "wb") as fd:
181 chunk = True
~/miniconda3/envs/intake-stac-gui/lib/python3.7/site-packages/aiohttp/client_reqrep.py in raise_for_status(self)
944 status=self.status,
945 message=self.reason,
--> 946 headers=self.headers)
947
948 def _cleanup_writer(self) -> None:
ClientResponseError: 401, message='Unauthorized', url=URL('https://urs.earthdata.nasa.gov/oauth/authorize?app_type=401&client_id=iwntGSgHy9yoog7Mjag0dQ&response_type=code&redirect_uri=https://grfn.asf.alaska.edu/door/oauth&state=aHR0cDovL2dyZm4uYXNmLmFsYXNrYS5lZHUvZG9vci9kb3dubG9hZC9TMS1HVU5XLUEtUi0xMzctdG9wcy0yMDIwMDMxMV8yMDIwMDMwNS0wMjAwMTMtNDMwNTVOXzQxNTE4Ti1QUC0wMGZlLXYyXzBfMi5uYw')
I also tried entering authentication settings via fsspec.config.conf['https'] = dict(client_kwargs={'auth': aiohttp.BasicAuth(username, password)}) but get the same traceback.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the issue with the fsspec.open examples, then inspect fsspec/implementations/cached.py and fsspec/implementations/http.py, which appear in the traceback. Trace how client_kwargs moves through the simplecache path and verify that the HTTP request receives the supplied authentication when the issue is fixed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- authentication, backend, networking
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100