GoogleCloudPlatform / GoogleCloudPlatform/gsutil

gsutil crashes when run in hadoop's long temp directory paths

Open
#228 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
918
Forks
335
PR merge metrics
No merged PRs in 30d

Description

I am attempting to use gsutil inside a hadoop streaming setup. When I try to run the following command (where I have preserved filename length but masked the names)

```
gsutil stat gs://XXXX-XX-XXXXX-XXXYXXX/XXYXXXXXYXXXXYXX
```

as my mapper I get the following error:

```
Process SyncManager-1:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python2.7/multiprocessing/managers.py", line 550, in _run_server
server = cls._Server(registry, address, authkey, serializer)
File "/usr/lib/python2.7/multiprocessing/managers.py", line 162, in __init__
self.listener = Listener(address=address, backlog=16)
File "/usr/lib/python2.7/multiprocessing/connection.py", line 132, in __init__
self._listener = SocketListener(address, family, backlog)
File "/usr/lib/python2.7/multiprocessing/connection.py", line 257, in __init__
self._socket.bind(address)
File "/usr/lib/python2.7/socket.py", line 224, in meth
return getattr(self._sock,name)(*args)
error: AF_UNIX path too long
```

It seems like gsutil might be creating temporary sockets in the current directory which overflow the UNIX limit when inside hadoop's long temporary paths.

My image is on `SMP Debian 3.14.15-2~bpo70+1` in case that is useful.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.