GoogleCloudPlatform / GoogleCloudPlatform/gsutil
'ForkAwareLocal' object has no attribute 'connection' for multithreaded cp
- Dominant language
- Python
- Stars
- 918
- Forks
- 335
- PR merge metrics
- No merged PRs in 30d
Description
When running `gsutil -m cp -r gs://example/ ./` on a fairly large folder on macOS with the system Python 3.7 (to prevent the issues from #961), I see many instances of the following error:
```
Exception in thread Thread-4:
Traceback (most recent call last):
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/multiprocessing/managers.py", line 788, in _callmethod
conn = self._tls.connection
AttributeError: 'ForkAwareLocal' object has no attribute 'connection'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/threading.py", line 917, in _bootstrap_inner
self.run()
File "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/gsutil/gslib/command.py", line 2348, in run
cls = copy.copy(class_map[caller_id])
File "", line 2, in __getitem__
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/multiprocessing/managers.py", line 792, in _callmethod
self._connect()
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/multiprocessing/managers.py", line 779, in _connect
conn = self._Client(self._token.address, authkey=self._authkey)
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/multiprocessing/connection.py", line 492, in Client
c = SocketClient(address)
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/multiprocessing/connection.py", line 619, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 61] Connection refused
```
As a result the command seems to hang after starting a first batch of downloads.
Contributor guide
Research direction
Start with the multithreaded copy path in gslib/command.py around the reported run location, then inspect how class_map is accessed from worker threads. Reproduce `gsutil -m cp -r` on macOS with Python 3.7 and trace the multiprocessing manager connection failure. Done means the command completes without hanging and the regression is covered by an existing or newly identified test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100