indygreg / indygreg/PyOxidizer

Method for building Rust projects with a simple `cargo build`

Open
#498 4 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
6.2k
Forks
256
PR merge metrics
No merged PRs in 30d

Description

My goal was to build a Rust project that embeds python and all dependencies via a simple `cargo build`. I finally figured out a method for doing this, but it's pretty much one big hack. Nevertheless, it may be helpful to others and could guide future development.

The problem with the project structure created by `pyoxidizer init-rust-project` is that the `pyo3-build-config` dependency needs `PYO3_CONFIG_FILE` to point to `pyo3-build-config-file.txt`, which is generated by `pyoxidizer run-build-script`. As a result, the only way to get a working build is to generate the artifacts first and then pass the relevant environment variables to `cargo build`, as described in #467. Running the generated `build.rs` in `build-mode-pyoxidizer-exe` mode does not work since all the `pyo3` dependencies are already built by that point using the default configuration.

A way to get everything to work with a one `cargo build` command is to have `pyo3-build-config` run `pyoxidizer run-build-script` directly. I used a [[patch] override](https://doc.rust-lang.org/cargo/reference/overriding-dependencies.html#the-patch-section) to do that. Essentially, I created my own `pyo3-build-config` package that runs `pyoxidizer`. It depends on the real `pyo3-build-config` to avoid re-implementing most of the functionality.

Below are the relevant file contents. I've omitted `./py/pyoxidizer.bzl` and other files generated by `pyoxidizer init-rust-project` because those are all standard. Use at your own risk.

Note that this still does not solve #466.

`./Cargo.toml`:
```toml
[workspace]
members = ["py"]
resolver = "2"

[patch.crates-io]
pyo3-build-config = { path = "py/pyo3-build-config" }
pyembed = { git = "https://github.com/indygreg/PyOxidizer.git", rev = "4cf22b8d76d6873471ba0fdf81e852fc8495955f" }
```

`./py/Cargo.toml`:
```toml
[package]
name = "py"
version = "0.1.0"
edition = "2021"
publish = false

[dependencies]
pyembed = { version = "0.20.0-pre", default-features = false }

# Required to pass DEP_PYEMBED_DEFAULT_CONFIG
pyo3-build-config = { path = "./pyo3-build-config", features = ["resolve-config"] }

[build-dependencies]
pyo3-build-config = { path = "./pyo3-build-config", features = ["resolve-config"] }
```

`./py/build.rs`:
```rust
fn main() {
println!("cargo:rerun-if-changed=build.rs");
println!(
"cargo:rustc-env=DEFAULT_PYTHON_CONFIG_RS={}",
std::env::var("DEP_PYEMBED_DEFAULT_CONFIG").expect(
"DEP_PYEMBED_DEFAULT_CONFIG not set (misconfigured pyo3-build-config wrapper?)"
)
);
}
```

`./py/pyo3-build-config/Cargo.toml`:
```toml
# This package should be added to `[patch.crates-io]` section in the root
# `Crate.toml`. This will cause all `pyo3` packages to depend on it. The build
# script verifies the dependency tree to ensure that everything is configured
# correctly.
#
# The `version` below is set to ensure that this package is prefered over the
# published versions, but it needs to be updated when a new minor release is
# made.
#
# This package depends on the real `pyo3-build-config` to avoid duplicating most
# of the functionality. The dependency versions must be kept in sync.
[package]
name = "pyo3-build-config"
version = "0.15.999"
description = "PyOxidizer-based wrapper around the real pyo3-build-config"
edition = "2021"
links = "pyembed"
publish = false

[dependencies]
once_cell = "1"
pyo3-build-config = { version = "0.15.0, <0.15.999", default-features = false }

[build-dependencies]
embed-resource = "1.6"
pyo3-build-config = { version = "0.15.0, <0.15.999", default-features = false }

[features]
resolve-config = []
```

`./py/pyo3-build-config/build.rs`:
```rust
use std::env::var_os;
use std::io::{BufRead, Cursor};
use std::path::{Path, PathBuf};
use std::process::Command;

fn main() {
if var_os("CARGO_FEATURE_RESOLVE_CONFIG").is_none() {
println!("cargo:warning=resolve-config feature not enabled; build script in no-op mode");
return;
}

ensure_dependency_tree();
run_build_script(var_os("PYOXIDIZER_CONFIG").unwrap_or_else(|| "../pyoxidizer.bzl".into()));

// TODO: Python source location should be configurable
println!("cargo:rerun-if-changed=../pysrc");
}

// Runs `pyoxidizer run-build-script`.
fn run_build_script(cfg: impl AsRef) {
let cfg = cfg.as_ref().canonicalize().unwrap();
let root = cfg.parent().unwrap();
let out_dir = PathBuf::from(var_os("OUT_DIR").unwrap());

// PyOxidizer compares the mtime of the pyoxidizer binary and pyoxidizer.bzl
// with that of default_python_config.rs to determine if the artifacts are
// current. We want cargo to be in charge of cache invalidation, so remove
// the config to rebuild the artifacts unconditionally.
let out_cfg = out_dir.join("default_python_config.rs");
let _ = std::fs::remove_file(&out_cfg);

println!("cargo:rerun-if-env-changed=PYOXIDIZER_EXE");
let status = Command::new(var_os("PYOXIDIZER_EXE").unwrap_or_else(|| "pyoxidizer".into()))
.args(["run-build-script", "--system-rust", "build.rs"])
.env("PYOXIDIZER_CONFIG", &cfg)
.env("PYOXIDIZER_ARTIFACT_DIR", &out_dir)
.current_dir(root)
.status()
.expect("failed to run pyoxidizer");
if !status.success() {
panic!("`pyoxidizer run-build-script` error: {}", status);
}

// Embed Windows resource file
if var_os("CARGO_CFG_TARGET_FAMILY").unwrap() == "windows" {
for f in std::fs::read_dir(root).unwrap() {
let f = f.unwrap();
if f.file_name().to_string_lossy().ends_with("-manifest.rc") {
embed_resource::compile(f.path());
}
}
}

// Parse the config file to guarantee that this will not fail at run time
pyo3_build_config::InterpreterConfig::from_path(out_dir.join("pyo3-build-config-file.txt"))
.expect("failed to parse contents of PYO3_CONFIG_FILE");
println!("cargo:default_config={}", out_cfg.to_str().unwrap());
}

/// Ensures that pyo3 is using the patched version of pyo3-build-config. This is
/// necessary because the `[patch]` mechanism does not provide a way of ignoring
/// package versions from crates.io, so the `version` in our `Cargo.toml` needs
/// to be set correctly in order for this package to shadow the official one.
fn ensure_dependency_tree() {
let out = Command::new(var_os("CARGO").unwrap())
.args([
"tree",
"--quiet",
"--workspace",
"--edges=normal",
"--invert",
concat!(env!("CARGO_PKG_NAME"), ":", env!("CARGO_PKG_VERSION")),
"--depth=3",
"--prefix=none",
"--format={lib}",
"--frozen",
])
.output()
.expect("failed to execute `cargo tree`")
.stdout;
if !Cursor::new(out).lines().any(|ln| ln.unwrap() == "pyo3") {
panic!(
"pyo3 is using the wrong {} package (check 'version' in {:?})",
env!("CARGO_PKG_NAME"),
PathBuf::from(var_os("CARGO_MANIFEST_DIR").unwrap()).join("Cargo.toml")
);
}
}
```

`./py/pyo3-build-config/src/lib.rs`:
```rust
//! PyOxidizer-based wrapper around the real pyo3-build-config.
//!
//! This crate forwards most of the functionality to the real pyo3-build-config,
//! but uses PyOxidizer-generated `pyo3-build-config-file.txt`.

pub use pyo3_build_config::*;

#[cfg(feature = "resolve-config")]
pub fn use_pyo3_cfgs() {
get().emit_pyo3_cfgs();
}

#[doc(hidden)]
#[cfg(feature = "resolve-config")]
pub fn get() -> &'static InterpreterConfig {
use once_cell::sync::OnceCell;
static CFG: OnceCell = OnceCell::new();
CFG.get_or_init(|| pyo3_build_script_impl::resolve_interpreter_config().unwrap())
}

#[doc(hidden)]
pub fn env_var(var: &str) -> Option {
if cfg!(feature = "resolve-config") {
println!("cargo:rerun-if-env-changed={}", var);
}
std::env::var_os(var)
}

#[doc(hidden)]
pub mod pyo3_build_script_impl {
pub use super::env_var;
pub use pyo3_build_config::pyo3_build_script_impl::*;

#[cfg(feature = "resolve-config")]
pub fn resolve_interpreter_config() -> errors::Result {
const CFG: &str = include_str!(concat!(env!("OUT_DIR"), "/pyo3-build-config-file.txt"));
InterpreterConfig::from_reader(std::io::Cursor::new(CFG))
}
}
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the generated ./Cargo.toml, ./py/Cargo.toml, ./py/build.rs, and the pyo3-build-config wrapper files shown in the issue, then compare the constraints described in #467 and #466. Done would require a defined supported approach for building the embedded Python project with one cargo build command, including agreement on the unresolved #466 behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
build-system, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.