aspect-build / aspect-build/rules_py
[Bug]: py_pex_binary causes some dependencies to conflict
- Dominant language
- Starlark
- Stars
- 145
- Forks
- 97
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 71
Description
### What happened?
I'm using Spark-NLP alongside hnswlib-spark in the same PySpark project. If I run the `py_binary` target directly it works fine but if I use the `py_pex_binary` for that target, I get "`ModuleNotFoundError: No module named 'com.johnsnowlabs'`". I've attached [a minimal repro of the problem](https://github.com/user-attachments/files/18507679/pex_repro.zip).
I've dug into the code of both dependencies and I see they both define a top-level `com` package. I wonder if this confuses the PEX build, even though it works fine as a regular python binary run. But I don't understand enough about how the PEX is built to confirm this.
### Version
Development (host) and target OS/architectures:
Output of `bazel --version`: `bazel 7.1.1`
Version of the Aspect rules, or other relevant rules from your
`WORKSPACE` or `MODULE.bazel` file:
aspect_rules_py - 1.1.0
rules_python - 0.37.1
Language(s) and/or frameworks involved:
- Python 3.9
- Pyspark
- Spark-NLP
- hnswlib-spark
### How to reproduce
```shell
Unpack the attached repro project. In the root run: `bazel run //:main`
Note it succeeds:
$ bazel run //:main
INFO: Invocation ID: 9f4f172c-0511-4ef0-a88b-0aec821e7321
INFO: Analyzed target //:main (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //:main up-to-date:
bazel-bin/main
bazel-bin/main.venv.pth
INFO: Elapsed time: 0.340s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/main
Success
Run `bazel run //:pex`
Note it fails:
$ bazel run //:pex
INFO: Invocation ID: 2a74ee90-6f47-480e-9158-42af27d2a145
INFO: Analyzed target //:pex (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //:pex up-to-date:
bazel-bin/pex.pex
INFO: Elapsed time: 0.330s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/pex.pex
Traceback (most recent call last):
File "/home/alex/.pyenv/versions/3.9.17/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/alex/.pyenv/versions/3.9.17/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/alex/.pex/unzipped_pexes/a052d8e4ed4967bf5d61d49ffd692b19bc15c5ce/__main__.py", line 119, in
bootstrap_pex(__entry_point__, execute=__execute__, venv_dir=__venv_dir__)
File "/home/alex/.pex/unzipped_pexes/a052d8e4ed4967bf5d61d49ffd692b19bc15c5ce/.bootstrap/pex/pex_bootstrapper.py", line 627, in bootstrap_pex
pex.PEX(entry_point).execute()
File "/home/alex/.pex/unzipped_pexes/a052d8e4ed4967bf5d61d49ffd692b19bc15c5ce/.bootstrap/pex/pex.py", line 562, in execute
sys.exit(self._wrap_coverage(self._wrap_profiling, self._execute))
File "/home/alex/.pex/unzipped_pexes/a052d8e4ed4967bf5d61d49ffd692b19bc15c5ce/.bootstrap/pex/pex.py", line 469, in _wrap_coverage
return runner(*args)
File "/home/alex/.pex/unzipped_pexes/a052d8e4ed4967bf5d61d49ffd692b19bc15c5ce/.bootstrap/pex/pex.py", line 500, in _wrap_profiling
return runner(*args)
File "/home/alex/.pex/unzipped_pexes/a052d8e4ed4967bf5d61d49ffd692b19bc15c5ce/.bootstrap/pex/pex.py", line 606, in _execute
return self.execute_entry(
File "/home/alex/.pex/unzipped_pexes/a052d8e4ed4967bf5d61d49ffd692b19bc15c5ce/.bootstrap/pex/pex.py", line 810, in execute_entry
return self.execute_module(entry_point.module)
File "/home/alex/.pex/unzipped_pexes/a052d8e4ed4967bf5d61d49ffd692b19bc15c5ce/.bootstrap/pex/pex.py", line 818, in execute_module
runpy.run_module(module_name, run_name="__main__", alter_sys=True)
File "/home/alex/.pyenv/versions/3.9.17/lib/python3.9/runpy.py", line 225, in run_module
return _run_module_code(code, init_globals, run_name, mod_spec)
File "/home/alex/.pyenv/versions/3.9.17/lib/python3.9/runpy.py", line 97, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "/home/alex/.pyenv/versions/3.9.17/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/alex/.pex/unzipped_pexes/a052d8e4ed4967bf5d61d49ffd692b19bc15c5ce/main.py", line 3, in
from sparknlp.annotator import DocumentAssembler, MPNetEmbeddings
File "/home/alex/.pex/installed_wheels/816263079eb751caef16984ce404fa5841e414fd/spark-nlp-5.5.1/sparknlp/__init__.py", line 19, in
from sparknlp import annotator
File "/home/alex/.pex/installed_wheels/816263079eb751caef16984ce404fa5841e414fd/spark-nlp-5.5.1/sparknlp/annotator/__init__.py", line 59, in
__import__("com.johnsnowlabs.nlp")
ModuleNotFoundError: No module named 'com.johnsnowlabs'
```
### Any other information?
_No response_
Contributor guide
Research direction
Unpack the attached pex_repro.zip and first run `bazel run //:main` and `bazel run //:pex` to reproduce the differing outcomes. Start by tracing the `py_pex_binary` target and the two dependencies that define the top-level `com` package; done means the PEX target runs successfully and imports `com.johnsnowlabs`.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- build-system
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100