fsspec / fsspec/filesystem_spec

HTTP client_kwargs authentication not used with simplecache

Open
#484 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.4k
Forks
490
Avg merge
2d 3h
Merged PRs (30d)
38

Description

Using fsspec='0.8.4+70.g33d532f' if a simplecache:: prefix is used with HTTP:// it seems that authentication settings are not used leading to ClientResponseError: 401, message='Unauthorized'

This works:

import fsspec
import xarray as xr
import netrc
import aiohttp

(username, account, password) = netrc.netrc().authenticators("urs.earthdata.nasa.gov")
uri = 'https://grfn.asf.alaska.edu/door/download/S1-GUNW-A-R-137-tops-20200311_20200305-020013-43055N_41518N-PP-00fe-v2_0_2.nc'
with fsspec.open(item.data.urlpath, 
                 client_kwargs={'auth': aiohttp.BasicAuth(username, password)}) as f:
    ds = xr.open_dataset(f, group='/science/grids/data', engine='h5netcdf', chunks={})

This does not:

with fsspec.open('simplecache::'+item.data.urlpath, 
                 client_kwargs={'auth': aiohttp.BasicAuth(username, password)}) as f:
    ds = xr.open_dataset(f, group='/science/grids/data', engine='h5netcdf', chunks={})
---------------------------------------------------------------------------
ClientResponseError                       Traceback (most recent call last)
<ipython-input-17-a30d2d9da865> in <module>
      1 # with simplecache
      2 with fsspec.open('simplecache::'+item.data.urlpath, 
----> 3                  client_kwargs={'auth': aiohttp.BasicAuth(username, password)}) as f:
      4     ds = xr.open_dataset(f, group='/science/grids/data', engine='h5netcdf', chunks={})
      5 

~/GitHub/filesystem_spec/fsspec/core.py in __enter__(self)
    100         mode = self.mode.replace("t", "").replace("b", "") + "b"
    101 
--> 102         f = self.fs.open(self.path, mode=mode)
    103 
    104         self.fobjects = [f]

~/GitHub/filesystem_spec/fsspec/implementations/cached.py in <lambda>(*args, **kw)
    384             # all the methods defined in this class. Note `open` here, since
    385             # it calls `_open`, but is actually in superclass
--> 386             return lambda *args, **kw: getattr(type(self), item)(self, *args, **kw)
    387         if item in ["__reduce_ex__"]:
    388             raise AttributeError

~/GitHub/filesystem_spec/fsspec/spec.py in open(self, path, mode, block_size, cache_options, **kwargs)
    912                 autocommit=ac,
    913                 cache_options=cache_options,
--> 914                 **kwargs
    915             )
    916             if not ac:

~/GitHub/filesystem_spec/fsspec/implementations/cached.py in <lambda>(*args, **kw)
    384             # all the methods defined in this class. Note `open` here, since
    385             # it calls `_open`, but is actually in superclass
--> 386             return lambda *args, **kw: getattr(type(self), item)(self, *args, **kw)
    387         if item in ["__reduce_ex__"]:
    388             raise AttributeError

~/GitHub/filesystem_spec/fsspec/implementations/cached.py in _open(self, path, mode, **kwargs)
    637                     f2.write(data)
    638         else:
--> 639             self.fs.get(path, fn)
    640         return self._open(path, mode)
    641 

~/GitHub/filesystem_spec/fsspec/asyn.py in get(self, rpath, lpath, recursive, **kwargs)
    279         lpaths = other_paths(rpaths, lpath)
    280         [os.makedirs(os.path.dirname(lp), exist_ok=True) for lp in lpaths]
--> 281         return sync(self.loop, self._get, rpaths, lpaths)
    282 
    283 

~/GitHub/filesystem_spec/fsspec/asyn.py in sync(loop, func, callback_timeout, *args, **kwargs)
     69     if error[0]:
     70         typ, exc, tb = error[0]
---> 71         raise exc.with_traceback(tb)
     72     else:
     73         return result[0]

~/GitHub/filesystem_spec/fsspec/asyn.py in f()
     53             if callback_timeout is not None:
     54                 future = asyncio.wait_for(future, callback_timeout)
---> 55             result[0] = await future
     56         except Exception:
     57             error[0] = sys.exc_info()

~/GitHub/filesystem_spec/fsspec/asyn.py in _get(self, rpaths, lpaths, **kwargs)
    267             *[
    268                 self._get_file(rpath, lpath, **kwargs)
--> 269                 for lpath, rpath in zip(lpaths, rpaths)
    270             ]
    271         )

~/GitHub/filesystem_spec/fsspec/implementations/http.py in _get_file(self, rpath, lpath, chunk_size, **kwargs)
    177             if r.status == 404:
    178                 raise FileNotFoundError(rpath)
--> 179             r.raise_for_status()
    180             with open(lpath, "wb") as fd:
    181                 chunk = True

~/miniconda3/envs/intake-stac-gui/lib/python3.7/site-packages/aiohttp/client_reqrep.py in raise_for_status(self)
    944                 status=self.status,
    945                 message=self.reason,
--> 946                 headers=self.headers)
    947 
    948     def _cleanup_writer(self) -> None:

ClientResponseError: 401, message='Unauthorized', url=URL('https://urs.earthdata.nasa.gov/oauth/authorize?app_type=401&client_id=iwntGSgHy9yoog7Mjag0dQ&response_type=code&redirect_uri=https://grfn.asf.alaska.edu/door/oauth&state=aHR0cDovL2dyZm4uYXNmLmFsYXNrYS5lZHUvZG9vci9kb3dubG9hZC9TMS1HVU5XLUEtUi0xMzctdG9wcy0yMDIwMDMxMV8yMDIwMDMwNS0wMjAwMTMtNDMwNTVOXzQxNTE4Ti1QUC0wMGZlLXYyXzBfMi5uYw')

I also tried entering authentication settings via fsspec.config.conf['https'] = dict(client_kwargs={'auth': aiohttp.BasicAuth(username, password)}) but get the same traceback.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the issue with the fsspec.open examples, then inspect fsspec/implementations/cached.py and fsspec/implementations/http.py, which appear in the traceback. Trace how client_kwargs moves through the simplecache path and verify that the HTTP request receives the supplied authentication when the issue is fixed.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
authentication, backend, networking
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.