GoogleCloudPlatform / GoogleCloudPlatform/gsutil
gsutil flag '-m' exception with Python 3.8
- Dominant language
- Python
- Stars
- 918
- Forks
- 335
- PR merge metrics
- No merged PRs in 30d
Description
gsutil works with Python 8 until you use the '-m' flag:
python -V
-> Python 3.8.3
Your current Cloud SDK version is: 304.0.0
```python
gsutil -m cp -r gs://foo-bar/* .
Traceback (most recent call last):
File "/Users/mi/tools/google-cloud-sdk/platform/gsutil/gsutil", line 21, in
gsutil.RunMain()
File "/Users/mi/tools/google-cloud-sdk/platform/gsutil/gsutil.py", line 123, in RunMain
sys.exit(gslib.__main__.main())
File "/Users/mi/tools/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 429, in main
return _RunNamedCommandAndHandleExceptions(
File "/Users/mi/tools/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 767, in _RunNamedCommandAndHandleExceptions
_HandleUnknownFailure(e)
File "/Users/mi/tools/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 625, in _RunNamedCommandAndHandleExceptions
return command_runner.RunNamedCommand(command_name,
File "/Users/mi/tools/google-cloud-sdk/platform/gsutil/gslib/command_runner.py", line 411, in RunNamedCommand
return_code = command_inst.RunCommand()
File "/Users/mi/tools/google-cloud-sdk/platform/gsutil/gslib/commands/cp.py", line 1190, in RunCommand
self.Apply(_CopyFuncWrapper,
File "/Users/mi/tools/google-cloud-sdk/platform/gsutil/gslib/command.py", line 1499, in Apply
self._ParallelApply(
File "/Users/mi/tools/google-cloud-sdk/platform/gsutil/gslib/command.py", line 1719, in _ParallelApply
self._CreateNewConsumerPool(process_count, thread_count,
File "/Users/mi/tools/google-cloud-sdk/platform/gsutil/gslib/command.py", line 1384, in _CreateNewConsumerPool
p.start()
File "/Users/mi/.pyenv/versions/3.8.3/lib/python3.8/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)
File "/Users/mi/.pyenv/versions/3.8.3/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "/Users/mi/.pyenv/versions/3.8.3/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
return Popen(process_obj)
File "/Users/mi/.pyenv/versions/3.8.3/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
super().__init__(process_obj)
File "/Users/mi/.pyenv/versions/3.8.3/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
self._launch(process_obj)
File "/Users/mi/.pyenv/versions/3.8.3/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
reduction.dump(process_obj, fp)
File "/Users/mi/.pyenv/versions/3.8.3/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle '_io.TextIOWrapper' object
```
Contributor guide
Research direction
Reproduce `gsutil -m cp -r gs://foo-bar/* .` with Python 3.8 and Cloud SDK 304.0.0, then start at `gslib/commands/cp.py` and `gslib/command.py`, especially `RunCommand`, `_ParallelApply`, and `_CreateNewConsumerPool`. Done means the command no longer raises the reported pickling exception under this environment.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100