fsspec / fsspec/gcsfs

unexceptionally long timeout

Open
#633 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
398
Forks
181
Avg merge
2d 10h
Merged PRs (30d)
36

Description

In my app, I'd like to timeout after a certain amount of seconds if my bucket cannot be reached, and fall back on a local copy of my file.

This is useful on deployments where egress to external sites is heavily firewalled.

However, passing in all the timeout information I could find by looking at the doc (notably GCSFileSystem.session_kwargs), and GCSFileSystem.timeout (even though it's not documented), does not solve the problem.

It hangs for a few minutes before throwing an exception.

I think the timeout is not passed everywhere it should, especially in GCSFile.
But modifying the code of GCSFile to accept a timeout as argument did not work either.

It hangs on this line especially: https://github.com/fsspec/gcsfs/blob/5cb4479b0ac9737ed99b906befc88c9250cbd21c/gcsfs/core.py#L1785

It runs in a asyncio loop, which know next to nothing about, so I can't dig.

The traceback is below, to reproduce, simply add an unroutable IP to /etc/hosts, and run the code snippet.

To reproduce
from aiohttp import ClientTimeout
import fsspec

timeout = 1
gcsfs_kwargs = {
    "session_kwargs": {"timeout": ClientTimeout(total=timeout)},
    "timeout": timeout,
}

filename = "gs://your-gcs-bucket/your-file.json"

filesystem, _ = fsspec.url_to_fs(filename, **gcsfs_kwargs)

with filesystem.open(filename) as f:
    print(f.read())

❯ cat /etc/hosts
127.0.0.1	localhost
10.255.255.1 storage.googleapis.com

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
Traceback
ConnectionAbortedError: SSL handshake is taking longer than 60.0 seconds: aborting the connection
  File "aiohttp/connector.py", line 1025, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs)
  File "uvloop/loop.pyx", line 2084, in create_connection
  File "uvloop/loop.pyx", line 2079, in uvloop.loop.Loop.create_connection

ClientConnectorError: Cannot connect to host storage.googleapis.com:443 ssl:default [None]
  File "gcsfs/retry.py", line 126, in retry_request
    return await func(*args, **kwargs)
  File "gcsfs/core.py", line 426, in _request
    async with self.session.request(
  File "aiohttp/client.py", line 1197, in __aenter__
    self._resp = await self._coro
  File "aiohttp/client.py", line 581, in _request
    conn = await self._connector.connect(
  File "aiohttp/connector.py", line 544, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "aiohttp/connector.py", line 944, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
  File "aiohttp/connector.py", line 1257, in _create_direct_connection
    raise last_exc
  File "aiohttp/connector.py", line 1226, in _create_direct_connection
    transp, proto = await self._wrap_create_connection(
  File "aiohttp/connector.py", line 1033, in _wrap_create_connection
    raise client_error(req.connection_key, exc) from exc

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the reproduction and inspect gcsfs/core.py around line 1785 and _request around line 426, then read gcsfs/retry.py around line 126 to trace timeout and retry handling. Confirm the timeout is honored for the connection attempt and that the example fails within the requested interval.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, python
Domain
backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.