apache / apache/beam

Proto generation installation is broken for Windows

Open
#18,579 0 comments 0 reactions 0 assignees View on GitHub
bug core P3 python windows
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

"python setup.py sdist" and "python setup.py install" raises following error for Windows. Failure happens when running gen_protos.py. Robert, seems like you added this script. Can you take a look ?

C:\Users\chamikara\beam_testing\windows_test_10_16_2017\beam\sdks\python\gen_protos.py:48: UserWarning: Installing grpcio-tools is recommended for development.
warnings.warn('Installing grpcio-tools is recommended for development.')
Traceback (most recent call last):
File "", line 1, in
File "c:\python27\Lib\multiprocessing\forking.py", line 380, in main
prepare(preparation_data)
File "c:\python27\Lib\multiprocessing\forking.py", line 510, in prepare
'__parents_main__', file, path_name, etc
File "C:\Users\chamikara\beam_testing\windows_test_10_16_2017\beam\sdks\python\setup.py", line 203, in
'test': generate_protos_first(test),
File "c:\python27\Lib\distutils\core.py", line 151, in setup
dist.run_commands()
File "c:\python27\Lib\distutils\dist.py", line 953, in run_commands
self.run_command(cmd)
File "c:\python27\Lib\distutils\dist.py", line 972, in run_command
cmd_obj.run()
File "C:\Users\chamikara\beam_testing\windows_test_10_16_2017\env1\lib\site-packages\setuptools\command\install.py", line 67, in run
self.do_egg_install()
File "C:\Users\chamikara\beam_testing\windows_test_10_16_2017\env1\lib\site-packages\setuptools\command\install.py", line 109, in do_egg_install
self.run_command('bdist_egg')
File "c:\python27\Lib\distutils\cmd.py", line 326, in run_command
self.distribution.run_command(command)
File "c:\python27\Lib\distutils\dist.py", line 972, in run_command
cmd_obj.run()
File "C:\Users\chamikara\beam_testing\windows_test_10_16_2017\env1\lib\site-packages\setuptools\command\bdist_egg.py", line 169, in run
cmd = self.call_command('install_lib', warn_dir=0)
File "C:\Users\chamikara\beam_testing\windows_test_10_16_2017\env1\lib\site-packages\setuptools\command\bdist_egg.py", line 155, in call_command
self.run_command(cmdname)
File "c:\python27\Lib\distutils\cmd.py", line 326, in run_command
self.distribution.run_command(command)
File "c:\python27\Lib\distutils\dist.py", line 972, in run_command
cmd_obj.run()
File "C:\Users\chamikara\beam_testing\windows_test_10_16_2017\env1\lib\site-packages\setuptools\command\install_lib.py", line 11, in run
self.build()
File "c:\python27\Lib\distutils\command\install_lib.py", line 109, in build
self.run_command('build_py')
File "c:\python27\Lib\distutils\cmd.py", line 326, in run_command
self.distribution.run_command(command)
File "c:\python27\Lib\distutils\dist.py", line 972, in run_command
cmd_obj.run()
File "C:\Users\chamikara\beam_testing\windows_test_10_16_2017\beam\sdks\python\setup.py", line 143, in run
gen_protos.generate_proto_files()
File "C:\Users\chamikara\beam_testing\windows_test_10_16_2017\beam\sdks\python\gen_protos.py", line 84, in generate_proto_files
p.start()
File "c:\python27\Lib\multiprocessing\process.py", line 130, in start
self._popen = Popen(self)
File "c:\python27\Lib\multiprocessing\forking.py", line 258, in __init__
cmd = get_command_line() **** [rhandle]
File "c:\python27\Lib\multiprocessing\forking.py", line 358, in get_command_line
is not going to be frozen to produce a Windows executable.''')
RuntimeError:
Attempt to start a new process before the current process
has finished its bootstrapping phase.

This probably means that you are on Windows and you have
forgotten to use the proper idiom in the main module:

if __name__ \== '__main__':
freeze_support()
...

The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce a Windows executable.
Traceback (most recent call last):
File "setup.py", line 203, in
'test': generate_protos_first(test),
File "c:\python27\Lib\distutils\core.py", line 151, in setup
dist.run_commands()
File "c:\python27\Lib\distutils\dist.py", line 953, in run_commands
self.run_command(cmd)
File "c:\python27\Lib\distutils\dist.py", line 972, in run_command
cmd_obj.run()
File "C:\Users\chamikara\beam_testing\windows_test_10_16_2017\env1\lib\site-packages\setuptools\command\install.py", line 67, in run
self.do_egg_install()
File "C:\Users\chamikara\beam_testing\windows_test_10_16_2017\env1\lib\site-packages\setuptools\command\install.py", line 109, in do_egg_install
self.run_command('bdist_egg')
File "c:\python27\Lib\distutils\cmd.py", line 326, in run_command
self.distribution.run_command(command)
File "c:\python27\Lib\distutils\dist.py", line 972, in run_command
cmd_obj.run()
File "C:\Users\chamikara\beam_testing\windows_test_10_16_2017\env1\lib\site-packages\setuptools\command\bdist_egg.py", line 169, in run
cmd = self.call_command('install_lib', warn_dir=0)
File "C:\Users\chamikara\beam_testing\windows_test_10_16_2017\env1\lib\site-packages\setuptools\command\bdist_egg.py", line 155, in call_command
self.run_command(cmdname)
File "c:\python27\Lib\distutils\cmd.py", line 326, in run_command
self.distribution.run_command(command)
File "c:\python27\Lib\distutils\dist.py", line 972, in run_command
cmd_obj.run()
File "C:\Users\chamikara\beam_testing\windows_test_10_16_2017\env1\lib\site-packages\setuptools\command\install_lib.py", line 11, in run
self.build()
File "c:\python27\Lib\distutils\command\install_lib.py", line 109, in build
self.run_command('build_py')
File "c:\python27\Lib\distutils\cmd.py", line 326, in run_command
self.distribution.run_command(command)
File "c:\python27\Lib\distutils\dist.py", line 972, in run_command
cmd_obj.run()
File "setup.py", line 143, in run
gen_protos.generate_proto_files()
File "C:\Users\chamikara\beam_testing\windows_test_10_16_2017\beam\sdks\python\gen_protos.py", line 87, in generate_proto_files
raise ValueError("Proto generation failed (see log for details).")
ValueError: Proto generation failed (see log for details).
(env1) PS C:\Users\chamikara\beam_testing\windows_test_10_16_2017\beam\sdks\python\>

Imported from Jira [BEAM-3065](https://issues.apache.org/jira/browse/BEAM-3065). Original Jira may contain additional context.
Reported by: chamikara.

Contributor guide

Open the contributing guide

Research direction

Start with sdks/python/gen_protos.py, especially generate_proto_files(), and sdks/python/setup.py where proto generation is invoked during installation. Reproduce the failure with python setup.py sdist or python setup.py install on Windows, then verify that installation completes and proto generation no longer raises the multiprocessing error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.