GoogleCloudPlatform / GoogleCloudPlatform/gsutil

gsutil flag '-m' exception with Python 3.8

Open
#1,085 3 comments 3 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
918
Forks
335
PR merge metrics
No merged PRs in 30d

Description

gsutil works with Python 8 until you use the '-m' flag:

python -V
-> Python 3.8.3

Your current Cloud SDK version is: 304.0.0

```python
gsutil -m cp -r gs://foo-bar/* .
Traceback (most recent call last):
File "/Users/mi/tools/google-cloud-sdk/platform/gsutil/gsutil", line 21, in
gsutil.RunMain()
File "/Users/mi/tools/google-cloud-sdk/platform/gsutil/gsutil.py", line 123, in RunMain
sys.exit(gslib.__main__.main())
File "/Users/mi/tools/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 429, in main
return _RunNamedCommandAndHandleExceptions(
File "/Users/mi/tools/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 767, in _RunNamedCommandAndHandleExceptions
_HandleUnknownFailure(e)
File "/Users/mi/tools/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 625, in _RunNamedCommandAndHandleExceptions
return command_runner.RunNamedCommand(command_name,
File "/Users/mi/tools/google-cloud-sdk/platform/gsutil/gslib/command_runner.py", line 411, in RunNamedCommand
return_code = command_inst.RunCommand()
File "/Users/mi/tools/google-cloud-sdk/platform/gsutil/gslib/commands/cp.py", line 1190, in RunCommand
self.Apply(_CopyFuncWrapper,
File "/Users/mi/tools/google-cloud-sdk/platform/gsutil/gslib/command.py", line 1499, in Apply
self._ParallelApply(
File "/Users/mi/tools/google-cloud-sdk/platform/gsutil/gslib/command.py", line 1719, in _ParallelApply
self._CreateNewConsumerPool(process_count, thread_count,
File "/Users/mi/tools/google-cloud-sdk/platform/gsutil/gslib/command.py", line 1384, in _CreateNewConsumerPool
p.start()
File "/Users/mi/.pyenv/versions/3.8.3/lib/python3.8/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)
File "/Users/mi/.pyenv/versions/3.8.3/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "/Users/mi/.pyenv/versions/3.8.3/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
return Popen(process_obj)
File "/Users/mi/.pyenv/versions/3.8.3/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
super().__init__(process_obj)
File "/Users/mi/.pyenv/versions/3.8.3/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
self._launch(process_obj)
File "/Users/mi/.pyenv/versions/3.8.3/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
reduction.dump(process_obj, fp)
File "/Users/mi/.pyenv/versions/3.8.3/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle '_io.TextIOWrapper' object
```

Contributor guide

Open the contributing guide

Research direction

Reproduce `gsutil -m cp -r gs://foo-bar/* .` with Python 3.8 and Cloud SDK 304.0.0, then start at `gslib/commands/cp.py` and `gslib/command.py`, especially `RunCommand`, `_ParallelApply`, and `_CreateNewConsumerPool`. Done means the command no longer raises the reported pickling exception under this environment.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.