unexceptionally long timeout
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 398
- Forks
- 181
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 36
Description
In my app, I'd like to timeout after a certain amount of seconds if my bucket cannot be reached, and fall back on a local copy of my file.
This is useful on deployments where egress to external sites is heavily firewalled.
However, passing in all the timeout information I could find by looking at the doc (notably GCSFileSystem.session_kwargs), and GCSFileSystem.timeout (even though it's not documented), does not solve the problem.
It hangs for a few minutes before throwing an exception.
I think the timeout is not passed everywhere it should, especially in GCSFile.
But modifying the code of GCSFile to accept a timeout as argument did not work either.
It hangs on this line especially: https://github.com/fsspec/gcsfs/blob/5cb4479b0ac9737ed99b906befc88c9250cbd21c/gcsfs/core.py#L1785
It runs in a asyncio loop, which know next to nothing about, so I can't dig.
The traceback is below, to reproduce, simply add an unroutable IP to /etc/hosts, and run the code snippet.
To reproduce
from aiohttp import ClientTimeout
import fsspec
timeout = 1
gcsfs_kwargs = {
"session_kwargs": {"timeout": ClientTimeout(total=timeout)},
"timeout": timeout,
}
filename = "gs://your-gcs-bucket/your-file.json"
filesystem, _ = fsspec.url_to_fs(filename, **gcsfs_kwargs)
with filesystem.open(filename) as f:
print(f.read())
❯ cat /etc/hosts
127.0.0.1 localhost
10.255.255.1 storage.googleapis.com
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
Traceback
ConnectionAbortedError: SSL handshake is taking longer than 60.0 seconds: aborting the connection
File "aiohttp/connector.py", line 1025, in _wrap_create_connection
return await self._loop.create_connection(*args, **kwargs)
File "uvloop/loop.pyx", line 2084, in create_connection
File "uvloop/loop.pyx", line 2079, in uvloop.loop.Loop.create_connection
ClientConnectorError: Cannot connect to host storage.googleapis.com:443 ssl:default [None]
File "gcsfs/retry.py", line 126, in retry_request
return await func(*args, **kwargs)
File "gcsfs/core.py", line 426, in _request
async with self.session.request(
File "aiohttp/client.py", line 1197, in __aenter__
self._resp = await self._coro
File "aiohttp/client.py", line 581, in _request
conn = await self._connector.connect(
File "aiohttp/connector.py", line 544, in connect
proto = await self._create_connection(req, traces, timeout)
File "aiohttp/connector.py", line 944, in _create_connection
_, proto = await self._create_direct_connection(req, traces, timeout)
File "aiohttp/connector.py", line 1257, in _create_direct_connection
raise last_exc
File "aiohttp/connector.py", line 1226, in _create_direct_connection
transp, proto = await self._wrap_create_connection(
File "aiohttp/connector.py", line 1033, in _wrap_create_connection
raise client_error(req.connection_key, exc) from exc
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the reproduction and inspect gcsfs/core.py around line 1785 and _request around line 426, then read gcsfs/retry.py around line 126 to trace timeout and retry handling. Confirm the timeout is honored for the connection attempt and that the example fails within the requested interval.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, python
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100