boostorg / boostorg/build

Better support for python packaging and virtualenvs

Open
#706 1 comment 0 reactions 0 assignees View on GitHub
transition
Dominant language
C++
Stars
251
Forks
63
PR merge metrics
No merged PRs in 30d

Description

I'm working on [libtorrent](https://github.com/arvidn/libtorrent/)'s python bindings and I'm finding that it's really difficult to package and test `b2` python targets using python ecosystem standards.

Python extensions are normally built by `distutils`, which always builds them against the running python interpreter. That is, if you build using `python3.6 setup.py bdist_wheel`, you'll get a wheel build against the `python3.6` you ran.

This dovetails with the standard python package testing workflow, which looks like this:
* Set up a virtual environment
* Build the package in that environment, for the environment
* Install the package
* Run tests against the installed code

Note that `distutils`' `bdist_wheel` is actually the only standard tooling to create python wheels. So if you want to package a python target from `b2` into a wheel, the most straightforward way is to write a `setup.py`, and customize its `build_ext` step to invoke `b2` instead of building using `distutils` logic.

So for both packaging and testing, we need to get `b2` to follow the python practice of _build for a given interpreter_. But `b2`'s design makes this hard.

We can try `b2 python=X.Y`, but we need to ensure this matches some `using python : X.Y : ... ` in some `*-config.jam`, which matches our given environment and not others, and isn't overridden by any other config.

The problem is that `b2` treats python like `gcc`, as some tool installed globally by a supervisor in one or more versions. But this is not how python works. Local, temporary virtual environments are the rule, not the exception. We certainly can't rely on `~/user-config.jam` to have what we need.

In libtorrent, I ended up having `setup.py` create a temporary `project-config.jam` like this:
```
import feature ;
feature.feature libtorrent-python : on ;
using python : X.Y : /path/to/my/pythonX.Y : ... : ... : on : ... ;
```
Then, `setup.py` invokes `b2 python=X.Y libtorrent-python=on ...`. The dummy `libtorrent-python` feature lets me select only my dynamically-created python configuration.

We're only halfway done, because we can't trust `python.jam`'s other configuration guesses.

I believe the code that guesses include and library search paths has never been tested in virtual environments. It doesn't honor the difference between `sys.exec_prefix` and `sys.base_exec_prefix`.

Worse than just being wrong on some platforms, the guesses appear kind of willy-nilly. For instance, it applies a `"pythonX.Y/config"` library search path on all non-windows platforms, but this appears to only be appropriate for cygwin.

Worse, the python config rule only allows a single library search path. There are certainly cases where we need multiple ones. I ended up resolving this by adding extra `library-path=...` requirements to my `b2` command. It's not really appropriate to apply these globally, but I couldn't think of another solution.

`distutils` also has code that guesses include and library paths, and it does a much better job. For our autogenerated python config, we override include and library paths with the `include_dirs` and `library_dirs` attributes of `distutils.command.build_ext.build_ext`. In case these are empty, I do not trust `python.jam`'s guesses not to be destructive, and I configure garbage search paths instead.

Finally, we want our `build_ext` to produce an artifact with name and location it normally would, so other `distutils` logic can consume it. `python.jam` is unhelpful again here, as its configuration guesses aren't helpful _and_ it appends a platform-specific suffix to whatever was configured. So if we get a correct value from `distutils`, we must anticipate `python.jam`'s mangling and de-mangle our value correctly.

I think in the short term:
* I should be able to invoke `b2 python.exe=/path/to/my/pythonX.Y`
* This should not require, and should disregard, any config in `*-config.jam`
* This should invoke the executable to discover include and library search paths, and configure the extension suffix, using logic directly from `build_ext` within `distutils`
* `python.jam`'s existing library and include search path guesses should be correct for virtual environments, on all platforms

In the long term, I think the boost team should come up with a more clear story for how a user can use `b2` python targets with standard python tools, and package them in standard ways. This was way too much work.

Contributor guide

Open the contributing guide

Research direction

Start by reading python.jam alongside the setup.py and distutils build_ext workflow described in the issue; compare them with the generated project-config.jam and existing *-config.jam configuration. Done means b2 can select a supplied Python executable without relying on those configs, discover virtualenv include and library paths, and produce the expected extension suffix across platforms.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
build-system, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.