aspect-build / aspect-build/rules_py

[Bug]: py_pex_binary causes some dependencies to conflict

Open
#495 0 comments 0 reactions 0 assignees View on GitHub
bug help wanted
Dominant language
Starlark
Stars
145
Forks
97
Avg merge
1d 1h
Merged PRs (30d)
71

Description

### What happened?

I'm using Spark-NLP alongside hnswlib-spark in the same PySpark project. If I run the `py_binary` target directly it works fine but if I use the `py_pex_binary` for that target, I get "`ModuleNotFoundError: No module named 'com.johnsnowlabs'`". I've attached [a minimal repro of the problem](https://github.com/user-attachments/files/18507679/pex_repro.zip).

I've dug into the code of both dependencies and I see they both define a top-level `com` package. I wonder if this confuses the PEX build, even though it works fine as a regular python binary run. But I don't understand enough about how the PEX is built to confirm this.

### Version

Development (host) and target OS/architectures:

Output of `bazel --version`: `bazel 7.1.1`

Version of the Aspect rules, or other relevant rules from your
`WORKSPACE` or `MODULE.bazel` file:

aspect_rules_py - 1.1.0
rules_python - 0.37.1

Language(s) and/or frameworks involved:
- Python 3.9
- Pyspark
- Spark-NLP
- hnswlib-spark

### How to reproduce

```shell
Unpack the attached repro project. In the root run: `bazel run //:main`
Note it succeeds:

$ bazel run //:main
INFO: Invocation ID: 9f4f172c-0511-4ef0-a88b-0aec821e7321
INFO: Analyzed target //:main (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //:main up-to-date:
bazel-bin/main
bazel-bin/main.venv.pth
INFO: Elapsed time: 0.340s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/main
Success

Run `bazel run //:pex`
Note it fails:

$ bazel run //:pex
INFO: Invocation ID: 2a74ee90-6f47-480e-9158-42af27d2a145
INFO: Analyzed target //:pex (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //:pex up-to-date:
bazel-bin/pex.pex
INFO: Elapsed time: 0.330s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/pex.pex
Traceback (most recent call last):
File "/home/alex/.pyenv/versions/3.9.17/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/alex/.pyenv/versions/3.9.17/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/alex/.pex/unzipped_pexes/a052d8e4ed4967bf5d61d49ffd692b19bc15c5ce/__main__.py", line 119, in
bootstrap_pex(__entry_point__, execute=__execute__, venv_dir=__venv_dir__)
File "/home/alex/.pex/unzipped_pexes/a052d8e4ed4967bf5d61d49ffd692b19bc15c5ce/.bootstrap/pex/pex_bootstrapper.py", line 627, in bootstrap_pex
pex.PEX(entry_point).execute()
File "/home/alex/.pex/unzipped_pexes/a052d8e4ed4967bf5d61d49ffd692b19bc15c5ce/.bootstrap/pex/pex.py", line 562, in execute
sys.exit(self._wrap_coverage(self._wrap_profiling, self._execute))
File "/home/alex/.pex/unzipped_pexes/a052d8e4ed4967bf5d61d49ffd692b19bc15c5ce/.bootstrap/pex/pex.py", line 469, in _wrap_coverage
return runner(*args)
File "/home/alex/.pex/unzipped_pexes/a052d8e4ed4967bf5d61d49ffd692b19bc15c5ce/.bootstrap/pex/pex.py", line 500, in _wrap_profiling
return runner(*args)
File "/home/alex/.pex/unzipped_pexes/a052d8e4ed4967bf5d61d49ffd692b19bc15c5ce/.bootstrap/pex/pex.py", line 606, in _execute
return self.execute_entry(
File "/home/alex/.pex/unzipped_pexes/a052d8e4ed4967bf5d61d49ffd692b19bc15c5ce/.bootstrap/pex/pex.py", line 810, in execute_entry
return self.execute_module(entry_point.module)
File "/home/alex/.pex/unzipped_pexes/a052d8e4ed4967bf5d61d49ffd692b19bc15c5ce/.bootstrap/pex/pex.py", line 818, in execute_module
runpy.run_module(module_name, run_name="__main__", alter_sys=True)
File "/home/alex/.pyenv/versions/3.9.17/lib/python3.9/runpy.py", line 225, in run_module
return _run_module_code(code, init_globals, run_name, mod_spec)
File "/home/alex/.pyenv/versions/3.9.17/lib/python3.9/runpy.py", line 97, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "/home/alex/.pyenv/versions/3.9.17/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/alex/.pex/unzipped_pexes/a052d8e4ed4967bf5d61d49ffd692b19bc15c5ce/main.py", line 3, in
from sparknlp.annotator import DocumentAssembler, MPNetEmbeddings
File "/home/alex/.pex/installed_wheels/816263079eb751caef16984ce404fa5841e414fd/spark-nlp-5.5.1/sparknlp/__init__.py", line 19, in
from sparknlp import annotator
File "/home/alex/.pex/installed_wheels/816263079eb751caef16984ce404fa5841e414fd/spark-nlp-5.5.1/sparknlp/annotator/__init__.py", line 59, in
__import__("com.johnsnowlabs.nlp")
ModuleNotFoundError: No module named 'com.johnsnowlabs'
```

### Any other information?

_No response_

Contributor guide

Open the contributing guide

Research direction

Unpack the attached pex_repro.zip and first run `bazel run //:main` and `bazel run //:pex` to reproduce the differing outcomes. Start by tracing the `py_pex_binary` target and the two dependencies that define the top-level `com` package; done means the PEX target runs successfully and imports `com.johnsnowlabs`.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.