danielgtaylor / danielgtaylor/python-betterproto
Generating multiple files [questions, bug? errors]
- Dominant language
- Python
- Stars
- 1.8k
- Forks
- 234
- PR merge metrics
- No merged PRs in 30d
Description
Hello, I'm new to GRPC (learning 5 days now).
My goal is to generate python SDK for [Gitlab Gitaly from published protos](https://gitlab.com/gitlab-org/gitaly/-/tree/master/proto).
I started with default `protoc` but quickly found out the problem with `sys.path` and everything else mentioned in motivation.
Then I found this project which gave me hope.
At first, I tried the latest release version `1.2.5` which lead `NoImplementedError -> Client Streaming`.
Then I installed [the latest master `5f7e4d58efa8aeb7af28c9ed89e4870548a1ce62` ](https://github.com/danielgtaylor/python-betterproto/commit/5f7e4d58efa8aeb7af28c9ed89e4870548a1ce62).
This let me process some files but not others.
Gitaly proto structure looks like this:
```
├── blob.proto
├── cleanup.proto
├── commit.proto
├── conflicts.proto
├── diff.proto
├── errors.proto
├── hook.proto
├── internal.proto
├── lint.proto
├── namespace.proto
├── objectpool.proto
├── operations.proto
├── praefect.proto
├── ref.proto
├── remote.proto
├── repository-service.proto
├── server.proto
├── shared.proto
├── smarthttp.proto
├── ssh.proto
├── transaction.proto
└── wiki.proto
```
Most of the files depend on `shared.proto` and `lint.proto`.
Here is what I tried:
## Default `protoc`:
```
python -m grpc_tools.protoc \
--proto_path=${PROTO_PATH} \
--python_out=${PROTO_OUTPUT}/normal \
--grpc_python_out=${PROTO_OUTPUT}/normal \
${PROTO_PATH}/ssh.proto
```
Result are 2 files that import `lint_pb2` and `shared_pb2` which are nowhere to be found... great
```
ssh_pb2.py
ssh_pb2_grpc.py
```
Then I tried the wildcard:
```
python -m grpc_tools.protoc \
--proto_path=${PROTO_PATH} \
--python_out=${PROTO_OUTPUT}/normal \
--grpc_python_out=${PROTO_OUTPUT}/normal \
${PROTO_PATH}/*.proto
```
This created all relevant files but was not really usable due to the broken `sys.path`.
## `betterprotoc`.
```
python -m grpc_tools.protoc \
--proto_path=${PROTO_PATH} \
--python_betterproto_out=${PROTO_OUTPUT}/better \
${PROTO_PATH}/ssh.proto
```
### Bug #1
The result is the generated package that does not respect the input filenames so everything ended up in `__init__.py` that was thousands of lines long.
Actual result:
```
better
├── gitaly
│ └── __init__.py
└── __init__.py
```
Expected result:
```
better
├── gitaly
│ ├── __init__.py
│ ├── shared.py
│ ├── lint.py
│ └── ssh.py
└── __init__.py
```
### Bug #2 failure with some files
I also tried to use the wildcard `*.proto`
```
python -m grpc_tools.protoc \
--proto_path=${PROTO_PATH} \
--python_betterproto_out=${PROTO_OUTPUT}/better \
${PROTO_PATH}/*.proto
```
But this resulted in a `StopIteration` I was not able to debug. What I realised was that this may be a problem only with some files.
I was able to pin down which files have caused this error:
```
transaction.proto
wiki.proto
repository-service.proto
ref.proto
diff.proto
commit.proto
```
Traceback shows something about `enum` type but that's unlikely because the other files that work also contain that type.
```
Traceback (most recent call last):
File "/home/user/cache/pypoetry/virtualenvs/gitaly-client-rXVxS8FM-py3.9/bin/protoc-gen-python_betterproto", line 8, in
sys.exit(main())
File "/home/user/cache/pypoetry/virtualenvs/gitaly-client-rXVxS8FM-py3.9/lib/python3.9/site-packages/betterproto/plugin/main.py", line 32, in main
response = generate_code(request)
File "/home/user/cache/pypoetry/virtualenvs/gitaly-client-rXVxS8FM-py3.9/lib/python3.9/site-packages/betterproto/plugin/parser.py", line 107, in generate_code
read_protobuf_service(service, index, output_package)
File "/home/user/cache/pypoetry/virtualenvs/gitaly-client-rXVxS8FM-py3.9/lib/python3.9/site-packages/betterproto/plugin/parser.py", line 191, in read_protobuf_service
ServiceMethodCompiler(
File "", line 7, in __init__
File "/home/user/cache/pypoetry/virtualenvs/gitaly-client-rXVxS8FM-py3.9/lib/python3.9/site-packages/betterproto/plugin/models.py", line 683, in __post_init__
self.mutable_default_args # ensure this is called before rendering
File "/home/user/cache/pypoetry/virtualenvs/gitaly-client-rXVxS8FM-py3.9/lib/python3.9/site-packages/betterproto/plugin/models.py", line 720, in mutable_default_args
and f.default_value_string != "None"
File "/home/user/cache/pypoetry/virtualenvs/gitaly-client-rXVxS8FM-py3.9/lib/python3.9/site-packages/betterproto/plugin/models.py", line 486, in default_value_string
enum = next(
StopIteration
--python_betterproto_out: protoc-gen-python_betterproto: Plugin failed with status code 1.
```
It would be very nice if `betterproto` can respect the initial `*.proto` structure because right now it seems to only read `package` IIUC.
Please let me know if I can help you with further debugging or this example is good enough.
my environment:
```
[tool.poetry.dependencies]
python = "~3.9"
grpcio = "^1.44.0"
grpcio-tools = "^1.44.0"
betterproto = {git = "https://github.com/danielgtaylor/python-betterproto.git", rev = "5f7e4d58efa8aeb7af28c9ed89e4870548a1ce62"}
[tool.poetry.dev-dependencies]
pytest = "^5.2"
ptpython = "^3.0.20"
types-protobuf = ">=0.1.14"
betterproto = {extras = ["compiler"], git = "https://github.com/danielgtaylor/python-betterproto.git", rev = "5f7e4d58efa8aeb7af28c9ed89e4870548a1ce62"}
```
Contributor guide
Assessment
This issue has not been assessed yet.