danielgtaylor / danielgtaylor/python-betterproto

Generating multiple files [questions, bug? errors]

Open
#357 0 comments 2 reactions 0 assignees View on GitHub
compiler-bug
Dominant language
Python
Stars
1.8k
Forks
234
PR merge metrics
No merged PRs in 30d

Description

Hello, I'm new to GRPC (learning 5 days now).
My goal is to generate python SDK for [Gitlab Gitaly from published protos](https://gitlab.com/gitlab-org/gitaly/-/tree/master/proto).
I started with default `protoc` but quickly found out the problem with `sys.path` and everything else mentioned in motivation.

Then I found this project which gave me hope.
At first, I tried the latest release version `1.2.5` which lead `NoImplementedError -> Client Streaming`.

Then I installed [the latest master `5f7e4d58efa8aeb7af28c9ed89e4870548a1ce62` ](https://github.com/danielgtaylor/python-betterproto/commit/5f7e4d58efa8aeb7af28c9ed89e4870548a1ce62).
This let me process some files but not others.

Gitaly proto structure looks like this:
```
├── blob.proto
├── cleanup.proto
├── commit.proto
├── conflicts.proto
├── diff.proto
├── errors.proto
├── hook.proto
├── internal.proto
├── lint.proto
├── namespace.proto
├── objectpool.proto
├── operations.proto
├── praefect.proto
├── ref.proto
├── remote.proto
├── repository-service.proto
├── server.proto
├── shared.proto
├── smarthttp.proto
├── ssh.proto
├── transaction.proto
└── wiki.proto
```

Most of the files depend on `shared.proto` and `lint.proto`.
Here is what I tried:
## Default `protoc`:
```
python -m grpc_tools.protoc \
--proto_path=${PROTO_PATH} \
--python_out=${PROTO_OUTPUT}/normal \
--grpc_python_out=${PROTO_OUTPUT}/normal \
${PROTO_PATH}/ssh.proto
```
Result are 2 files that import `lint_pb2` and `shared_pb2` which are nowhere to be found... great
```
ssh_pb2.py
ssh_pb2_grpc.py
```

Then I tried the wildcard:
```
python -m grpc_tools.protoc \
--proto_path=${PROTO_PATH} \
--python_out=${PROTO_OUTPUT}/normal \
--grpc_python_out=${PROTO_OUTPUT}/normal \
${PROTO_PATH}/*.proto
```

This created all relevant files but was not really usable due to the broken `sys.path`.

## `betterprotoc`.
```
python -m grpc_tools.protoc \
--proto_path=${PROTO_PATH} \
--python_betterproto_out=${PROTO_OUTPUT}/better \
${PROTO_PATH}/ssh.proto
```

### Bug #1
The result is the generated package that does not respect the input filenames so everything ended up in `__init__.py` that was thousands of lines long.

Actual result:
```
better
├── gitaly
│   └── __init__.py
└── __init__.py
```

Expected result:
```
better
├── gitaly
│   ├── __init__.py
│   ├── shared.py
│   ├── lint.py
│   └── ssh.py
└── __init__.py
```

### Bug #2 failure with some files
I also tried to use the wildcard `*.proto`
```
python -m grpc_tools.protoc \
--proto_path=${PROTO_PATH} \
--python_betterproto_out=${PROTO_OUTPUT}/better \
${PROTO_PATH}/*.proto
```
But this resulted in a `StopIteration` I was not able to debug. What I realised was that this may be a problem only with some files.

I was able to pin down which files have caused this error:
```
transaction.proto
wiki.proto
repository-service.proto
ref.proto
diff.proto
commit.proto
```

Traceback shows something about `enum` type but that's unlikely because the other files that work also contain that type.
```
Traceback (most recent call last):
File "/home/user/cache/pypoetry/virtualenvs/gitaly-client-rXVxS8FM-py3.9/bin/protoc-gen-python_betterproto", line 8, in
sys.exit(main())
File "/home/user/cache/pypoetry/virtualenvs/gitaly-client-rXVxS8FM-py3.9/lib/python3.9/site-packages/betterproto/plugin/main.py", line 32, in main
response = generate_code(request)
File "/home/user/cache/pypoetry/virtualenvs/gitaly-client-rXVxS8FM-py3.9/lib/python3.9/site-packages/betterproto/plugin/parser.py", line 107, in generate_code
read_protobuf_service(service, index, output_package)
File "/home/user/cache/pypoetry/virtualenvs/gitaly-client-rXVxS8FM-py3.9/lib/python3.9/site-packages/betterproto/plugin/parser.py", line 191, in read_protobuf_service
ServiceMethodCompiler(
File "", line 7, in __init__
File "/home/user/cache/pypoetry/virtualenvs/gitaly-client-rXVxS8FM-py3.9/lib/python3.9/site-packages/betterproto/plugin/models.py", line 683, in __post_init__
self.mutable_default_args # ensure this is called before rendering
File "/home/user/cache/pypoetry/virtualenvs/gitaly-client-rXVxS8FM-py3.9/lib/python3.9/site-packages/betterproto/plugin/models.py", line 720, in mutable_default_args
and f.default_value_string != "None"
File "/home/user/cache/pypoetry/virtualenvs/gitaly-client-rXVxS8FM-py3.9/lib/python3.9/site-packages/betterproto/plugin/models.py", line 486, in default_value_string
enum = next(
StopIteration
--python_betterproto_out: protoc-gen-python_betterproto: Plugin failed with status code 1.
```

It would be very nice if `betterproto` can respect the initial `*.proto` structure because right now it seems to only read `package` IIUC.
Please let me know if I can help you with further debugging or this example is good enough.

my environment:
```
[tool.poetry.dependencies]
python = "~3.9"
grpcio = "^1.44.0"
grpcio-tools = "^1.44.0"
betterproto = {git = "https://github.com/danielgtaylor/python-betterproto.git", rev = "5f7e4d58efa8aeb7af28c9ed89e4870548a1ce62"}

[tool.poetry.dev-dependencies]
pytest = "^5.2"
ptpython = "^3.0.20"
types-protobuf = ">=0.1.14"
betterproto = {extras = ["compiler"], git = "https://github.com/danielgtaylor/python-betterproto.git", rev = "5f7e4d58efa8aeb7af28c9ed89e4870548a1ce62"}
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.