fsspec / fsspec/filesystem_spec
Defining `timeout` as a YAML serializable object as part of fsspec storage_options instead of an `aiohttp.ClientTimeout` object
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 490
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 38
Description
Currently, the only way (as far as I know) to specify timeout as part of storage_options requires instantiating an aiohttp.ClientTimeout object
In [46]: import aiohttp, fsspec
In [49]: timeout = aiohttp.ClientTimeout(total=1000) # The maximal number of seconds
In [57]: storage_options={'http': {'client_kwargs': {'timeout': timeout}}, 'simplecache': {'same_names': True, 'cache_storage': "{{ env(INTAKE_LOCAL_CACHE_DIR) }}"}}
I would like to be able to specify the timeout in storage_options as a YAML serializable object and have fsspec instantiate the timeout object properly. I was wondering if this is something fsspec allows already and/or whether I am missing something?
Context
I would like to be able to specify the timeout as part of storage_options in an intake-catalog
sources:
fgapo_CarbonScope_apo99X_v2021:
driver: netcdf
description: "Using an atmospheric inversion technique similar to the Jena CO2 inversion, sea-air oxygen exchanges have been estimated from atmospheric O2/N2 measurements (Rödenbeck et al., 2008). In order to eliminate the influence of land processes on atmospheric oxygen abundance, the tracer Atmospheric Potential Oxygen (APO) has been used (Stephens et al., 1998). DOI: 10.17871/CarboScope-apo99X_v2021"
args:
urlpath: simplecache::https://www.bgc-jena.mpg.de/CarboScope/apo/INVERSION/OUTPUT/apo99X_v2021_daily.nc
xarray_kwargs:
decode_times: false
chunks: {}
storage_options:
client_kwargs:
timeout:
total: 3600
simplecache:
same_names: true
cache_storage: "{{ env(INTAKE_LOCAL_CACHE_DIR) }}"
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing how storage_options and client_kwargs reach the HTTP filesystem, with the YAML example and aiohttp.ClientTimeout behavior as the reference. Done means a YAML-serializable timeout mapping can be supplied through storage_options and produces the intended timeout without requiring callers to instantiate aiohttp.ClientTimeout.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- networking
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100