Cannot connect to host when running in dataproc
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 398
- Forks
- 181
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 36
Description
When running in dataproc, I get the following error:
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host storage.googleapis.com:443 ssl:default [Network is unreachable]
I have tried using both None, "cloud", and "google_default" for token. (I believe "cloud" is the correct option but not 100% sure). I am able to successfully read from the same bucket using the standard google.cloud.storage.Client, so authentication works there at least. I'm unsure what gcsfs is doing differently that would be causing this. I'm guessing I can fix this with some sort of dataproc config (firewall?), but since the standard gs client works, I'm wonderings why gcsfs doesn't.
Any help would be appreciated. Thanks!
Stack trace below:
Traceback (most recent call last):
File "/lib/python3.7/site-packages/aiohttp/connector.py", line 986, in _wrap_create_connection
return await self._loop.create_connection(*args, **kwargs) # type: ignore[return-value] # noqa
File "lib/python3.7/asyncio/base_events.py", line 962, in create_connection
raise exceptions[0]
File "/lib/python3.7/asyncio/base_events.py", line 949, in create_connection
await self.sock_connect(sock, address)
File "/lib/python3.7/asyncio/selector_events.py", line 473, in sock_connect
return await fut
File "/lib/python3.7/asyncio/selector_events.py", line 478, in _sock_connect
sock.connect(address)
OSError: [Errno 101] Network is unreachable
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/lib/python3.7/site-packages/gcsfs/retry.py", line 115, in retry_request
return await func(*args, **kwargs)
File "/lib/python3.7/site-packages/gcsfs/core.py", line 370, in _request
timeout=self.requests_timeout,
File "/lib/python3.7/site-packages/aiohttp/client.py", line 1138, in __aenter__
self._resp = await self._coro
File "/lib/python3.7/site-packages/aiohttp/client.py", line 536, in _request
req, traces=traces, timeout=real_timeout
File "/lib/python3.7/site-packages/aiohttp/connector.py", line 542, in connect
proto = await self._create_connection(req, traces, timeout)
File "/lib/python3.7/site-packages/aiohttp/connector.py", line 907, in _create_connection
_, proto = await self._create_direct_connection(req, traces, timeout)
File "/lib/python3.7/site-packages/aiohttp/connector.py", line 1206, in _create_direct_connection
raise last_exc
File "/lib/python3.7/site-packages/aiohttp/connector.py", line 1187, in _create_direct_connection
client_error=client_error,
File "/lib/python3.7/site-packages/aiohttp/connector.py", line 992, in _wrap_create_connection
raise client_error(req.connection_key, exc) from exc
This eventually causes a OSError: Passed non-file path: gs:// for the file I was trying to retrieve
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with gcsfs/core.py at _request and gcsfs/retry.py at retry_request, the entry points named in the stack trace. Reproduce the Dataproc request and compare it with the working google.cloud.storage.Client path; done means identifying whether the failure is in gcsfs or the Dataproc network configuration and documenting an actionable fix.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100