apache / apache/beam

gen_protos doesn't retry after failure

Open
#20,387 0 comments 0 reactions 0 assignees View on GitHub
bug core P3 python
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

[~chadrik]

While running using an outdated version of the protobuf package, I got this error:
```

$ python setup.py sdist
...
Traceback (most recent call last):
File "setup.py", line 339, in

'mypy': generate_protos_first(mypy),
File "/usr/local/google/home/ehudm/.pyenv/versions/3.6.9/envs/beam-py3.6.9/lib/python3.6/site-packages/setuptools/__init__.py",
line 145, in setup
return distutils.core.setup(**attrs)
File "/usr/local/google/home/ehudm/.pyenv/versions/3.6.9/lib/python3.6/distutils/core.py",
line 148, in setup
dist.run_commands()
File "/usr/local/google/home/ehudm/.pyenv/versions/3.6.9/lib/python3.6/distutils/dist.py",
line 955, in run_commands
self.run_command(cmd)
File "/usr/local/google/home/ehudm/.pyenv/versions/3.6.9/lib/python3.6/distutils/dist.py",
line 974, in run_command
cmd_obj.run()
File "/usr/local/google/home/ehudm/.pyenv/versions/3.6.9/envs/beam-py3.6.9/lib/python3.6/site-packages/setuptools/command/sdist.py",
line 44, in run
self.run_command('egg_info')
File "/usr/local/google/home/ehudm/.pyenv/versions/3.6.9/lib/python3.6/distutils/cmd.py",
line 313, in run_command
self.distribution.run_command(command)
File "/usr/local/google/home/ehudm/.pyenv/versions/3.6.9/lib/python3.6/distutils/dist.py",
line 974, in run_command
cmd_obj.run()
File "setup.py", line 253, in run
gen_protos.generate_proto_files()

File "/usr/local/google/home/ehudm/src/beam/sdks/python/gen_protos.py", line 345, in generate_proto_files

generate_urn_files(log, out_dir)
File "/usr/local/google/home/ehudm/src/beam/sdks/python/gen_protos.py",
line 188, in generate_urn_files
beam_runner_api_pb2 = _import('beam_runner_api_pb2')
File "/usr/local/google/home/ehudm/src/beam/sdks/python/gen_protos.py",
line 185, in _import
return __import__('api.%s' % m, fromlist=[None])
File "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/portability/api/beam_runner_api_pb2.py",
line 15, in
from . import endpoints_pb2 as endpoints__pb2
File "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/portability/api/endpoints_pb2.py",
line 21, in
create_key=_descriptor._internal_create_key,
AttributeError: module 'google.protobuf.descriptor'
has no attribute '_internal_create_key'

```

Rerunning sdist was successful, but the generated files were missing things like TRUNCATE_SIZED_RESTRICTION.

Issue for protobuf version: https://issues.apache.org/jira/browse/BEAM-10552

Imported from Jira [BEAM-10553](https://issues.apache.org/jira/browse/BEAM-10553). Original Jira may contain additional context.
Reported by: udim.

Contributor guide

Open the contributing guide

Research direction

Start with sdks/python/gen_protos.py, especially generate_proto_files and the import path shown in the traceback, then reproduce the failure through setup.py sdist using the outdated protobuf version. Done means a failed generation does not leave incomplete generated files and a subsequent run produces files containing items such as TRUNCATE_SIZED_RESTRICTION.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.