fsspec / fsspec/gcsfs

Cannot connect to host when running in dataproc

Open
#444 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
398
Forks
181
Avg merge
2d 10h
Merged PRs (30d)
36

Description

When running in dataproc, I get the following error:

aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host storage.googleapis.com:443 ssl:default [Network is unreachable]
I have tried using both None, "cloud", and "google_default" for token. (I believe "cloud" is the correct option but not 100% sure). I am able to successfully read from the same bucket using the standard google.cloud.storage.Client, so authentication works there at least. I'm unsure what gcsfs is doing differently that would be causing this. I'm guessing I can fix this with some sort of dataproc config (firewall?), but since the standard gs client works, I'm wonderings why gcsfs doesn't.

Any help would be appreciated. Thanks!

Stack trace below:

Traceback (most recent call last):
  File "/lib/python3.7/site-packages/aiohttp/connector.py", line 986, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs)  # type: ignore[return-value]  # noqa
  File "lib/python3.7/asyncio/base_events.py", line 962, in create_connection
    raise exceptions[0]
  File "/lib/python3.7/asyncio/base_events.py", line 949, in create_connection
    await self.sock_connect(sock, address)
  File "/lib/python3.7/asyncio/selector_events.py", line 473, in sock_connect
    return await fut
  File "/lib/python3.7/asyncio/selector_events.py", line 478, in _sock_connect
    sock.connect(address)
OSError: [Errno 101] Network is unreachable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/lib/python3.7/site-packages/gcsfs/retry.py", line 115, in retry_request
    return await func(*args, **kwargs)
  File "/lib/python3.7/site-packages/gcsfs/core.py", line 370, in _request
    timeout=self.requests_timeout,
  File "/lib/python3.7/site-packages/aiohttp/client.py", line 1138, in __aenter__
    self._resp = await self._coro
  File "/lib/python3.7/site-packages/aiohttp/client.py", line 536, in _request
    req, traces=traces, timeout=real_timeout
  File "/lib/python3.7/site-packages/aiohttp/connector.py", line 542, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/lib/python3.7/site-packages/aiohttp/connector.py", line 907, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
  File "/lib/python3.7/site-packages/aiohttp/connector.py", line 1206, in _create_direct_connection
    raise last_exc
  File "/lib/python3.7/site-packages/aiohttp/connector.py", line 1187, in _create_direct_connection
    client_error=client_error,
  File "/lib/python3.7/site-packages/aiohttp/connector.py", line 992, in _wrap_create_connection
    raise client_error(req.connection_key, exc) from exc

This eventually causes a OSError: Passed non-file path: gs:// for the file I was trying to retrieve

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with gcsfs/core.py at _request and gcsfs/retry.py at retry_request, the entry points named in the stack trace. Reproduce the Dataproc request and compare it with the working google.cloud.storage.Client path; done means identifying whether the failure is in gcsfs or the Dataproc network configuration and documenting an actionable fix.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.