indygreg / indygreg/PyOxidizer
Failing to build
- Dominant language
- Rust
- Stars
- 6.2k
- Forks
- 256
- PR merge metrics
- No merged PRs in 30d
Description
Hello
I'm attempting to package a pypi project `policy_sentry` https://github.com/salesforce/policy_sentry
I've installed pyoxidizer into a docker container:
```dockerfile
ARG PYTHON_MAJOR_VERSION=3.7
# Download Terraform binary
FROM debian:buster-20191224-slim as pyoxidizer-base
ARG PYTHON_MAJOR_VERSION
RUN apt-get update
RUN apt-get install -y curl=7.64.0-4+deb10u1
RUN apt-get install -y unzip=6.0-23+deb10u1
RUN apt-get install -y zip=3.0-11+b1
RUN apt-get install -y python3=${PYTHON_MAJOR_VERSION}.3-1
RUN apt-get install -y python3-pip=18.1-5
RUN apt-get install -y python3-venv=3.7.3-1
RUN apt-get install -y build-essential=12.6
RUN apt-get install -y pkg-config=0.29-6
RUN apt-get install -y libssl-dev=1.1.1d-0+deb10u2
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- --default-toolchain stable -y
RUN /root/.cargo/bin/cargo install pyoxidizer
WORKDIR /workspace
ADD policy_sentry .
CMD ["bash"]
```
My config file:
```
# This file defines how PyOxidizer application building and packaging is
# performed. See the pyoxidizer crate's documentation for extensive
# documentation on this file format.
# Obtain the default PythonDistribution for our build target. We link
# this distribution into our produced executable and extract the Python
# standard library from it.
def make_dist():
return default_python_distribution()
def make_macos_dist():
return PythonDistribution(
sha256="b46a861c05cb74b5b668d2ce44dcb65a449b9fef98ba5d9ec6ff6937829d5eec",
url="https://github.com/indygreg/python-build-standalone/releases/download/20190505/cpython-3.7.3-macos-20190506T0054.tar.zst"
)
def make_linux_dist():
return PythonDistribution(
sha256="608871543e6d2cb80e958638e31158355c578c114e12c77765ea5fb996a5a2c2",
local_path="/workspace/build/python_distributions/cpython-3.7.5-linux64-20191025T0506.tar.zst"
)
# Configuration files consist of functions which define build "targets."
# This function creates a Python executable and installs it in a destination
# directory.
def make_exe(dist):
# This variable defines the configuration of the
# embedded Python interpreter.
python_config = PythonInterpreterConfig(
# bytes_warning=0,
# dont_write_bytecode=True,
# ignore_environment=True,
# inspect=False,
# interactive=False,
# isolated=False,
# legacy_windows_fs_encoding=False,
# legacy_windows_stdio=False,
# no_site=True,
# no_user_site_directory=True,
# optimize_level=0,
# parser_debug=False,
# stdio_encoding=None,
# unbuffered_stdio=False,
# filesystem_importer=False,
# sys_frozen=False,
# sys_meipass=False,
# sys_paths=None,
# raw_allocator=None,
# terminfo_resolution="dynamic",
# terminfo_dirs=None,
# use_hash_seed=False,
# verbose=0,
# write_modules_directory_env=None,
# run_eval=None,
# run_module=None,
# run_noop=False,
# run_repl=True,
)
# The run_eval, run_module, run_noop, and run_repl arguments are mutually
# exclusive controls over what the interpreter should do once it initializes.
#
# run_eval -- Run the specified string value via `eval()`.
# run_module -- Import the specified module as __main__ and run it.
# run_noop -- Do nothing.
# run_repl -- Start a Python REPL.
#
# These arguments can be ignored if you are providing your own Rust code for
# starting the interpreter, as Rust code has full control over interpreter
# behavior.
# Produce a PythonExecutable from a Python distribution, embedded
# resources, and other options. The returned object represents the
# standalone executable that will be built.
exe = dist.to_python_executable(
name="policy_sentry",
config=python_config,
# Embed all extension modules, making this a fully-featured Python.
extension_module_filter='all',
# Only package the minimal set of extension modules needed to initialize
# a Python interpreter. Many common packages in Python's standard
# library won't work with this setting.
#extension_module_filter='minimal',
# Only package extension modules that don't require linking against
# non-Python libraries. e.g. will exclude support for OpenSSL, SQLite3,
# other features that require external libraries.
#extension_module_filter='no-libraries',
# Only package extension modules that don't link against GPL licensed
# libraries.
#extension_module_filter='no-gpl',
# Include Python module sources. This isn't strictly required and it does
# make binary sizes larger. But having the sources can be useful for
# activities such as debugging.
include_sources=True,
# Whether to include non-module resource data/files.
include_resources=False,
# Do not include functionality for testing Python itself.
include_test=False,
)
# Invoke `pip install` with our Python distribution to install a single package.
# `pip_install()` returns objects representing installed files.
# `add_python_resources()` adds these objects to our embedded context.
# exe.add_python_resources(dist.pip_install(["policy_sentry"]))
# Invoke `pip install` using a requirements file and add the collected files
# to our embedded context.
#exe.add_python_resources(dist.pip_install(["-r", "requirements.txt"]))
# Read Python files from a local directory and add them to our embedded
# context, taking just the resources belonging to the `foo` and `bar`
# Python packages.
#exe.add_python_resources(dist.read_package_root(
# path="/src/mypackage",
# packages=["foo", "bar"],
#)
# Discover Python files from a virtualenv and add them to our embedded
# context.
#exe.add_python_resources(dist.read_virtualenv(path="/path/to/venv"))
# Filter all resources collected so far through a filter of names
# in a file.
#exe.filter_from_files(files=["/path/to/filter-file"]))
# Return our `PythonExecutable` instance so it can be built and
# referenced by other consumers of this target.
return exe
def make_embedded_data(exe):
return exe.to_embedded_data()
def make_install(exe):
# Create an object that represents our installed application file layout.
files = FileManifest()
# Add the generated executable to our install layout in the root directory.
files.add_python_resource(".", exe)
return files
# Tell PyOxidizer about the build targets defined above.
register_target("dist", make_linux_dist)
register_target("exe", make_exe, depends=["dist"], default=True)
register_target("embedded", make_embedded_data, depends=["exe"], default_build_script=True)
register_target("install", make_install, depends=["exe"])
# Resolve whatever targets the invoker of this configuration file is requesting
# be resolved.
resolve_targets()
# END OF COMMON USER-ADJUSTED SETTINGS.
#
# Everything below this is typically managed by PyOxidizer and doesn't need
# to be updated by people.
PYOXIDIZER_VERSION = "0.6.0"
PYOXIDIZER_COMMIT = "UNKNOWN"
```
The build command fails with the line:
`register_target("dist", make_dist)`
```
root@ca4d128e7995:/workspace# /root/.cargo/bin/pyoxidizer build
resolving 1 targets
resolving target exe
resolving target dist
resolving Python distribution Url { url: "https://github.com/indygreg/python-build-standalone/releases/download/20191025/cpython-3.7.5-linux64-20191025T0506.tar.zst", sha256: "608871543e6d2cb80e958638e31158355c578c114e12c77765ea5fb996a5a2c2" }
Python distribution available at /workspace/./build/python_distributions/cpython-3.7.5-linux64-20191025T0506.tar.zst
reading data from Python distribution...
error: No such file or directory (os error 2)
```
I tried to build my own python standalone, and that succeeded with some tweaking, but fails to build for the same reason:
`register_target("dist", make_linux_dist)`
```
root@ca4d128e7995:/workspace# /root/.cargo/bin/pyoxidizer build
resolving 1 targets
resolving target exe
resolving target dist
resolving Python distribution Local { local_path: "/workspace/build/python_distributions/cpython-3.7.5-linux64-20191025T0506.tar.zst", sha256: "608871543e6d2cb80e958638e31158355c578c114e12c77765ea5fb996a5a2c2" }
existing /workspace/./build/python_distributions/cpython-3.7.5-linux64-20191025T0506.tar.zst passes SHA-256 integrity check
Python distribution available at /workspace/./build/python_distributions/cpython-3.7.5-linux64-20191025T0506.tar.zst
reading data from Python distribution...
error: No such file or directory (os error 2)
```
Attempting the macOS target fails also:
`register_target("dist", make_macos_dist)`
```
root@ca4d128e7995:/workspace# /root/.cargo/bin/pyoxidizer build
resolving 1 targets
resolving target exe
resolving target dist
resolving Python distribution Url { url: "https://github.com/indygreg/python-build-standalone/releases/download/20190505/cpython-3.7.3-macos-20190506T0054.tar.zst", sha256: "b46a861c05cb74b5b668d2ce44dcb65a449b9fef98ba5d9ec6ff6937829d5eec" }
Python distribution available at /workspace/./build/python_distributions/cpython-3.7.3-macos-20190506T0054.tar.zst
reading data from Python distribution...
error[PYOXIDIZER_BUILD]: Exec format error (os error 8)
--> ./pyoxidizer.bzl:75:11
|
75 | exe = dist.to_python_executable(
| ___________^
76 | | name="policy_sentry",
77 | | config=python_config,
78 | | # Embed all extension modules, making this a fully-featured Python.
... |
104 | | include_test=False,
105 | | )
| |_____^ to_python_executable()
error: Exec format error (os error 8)
```
And building a python-standalone dist for macOS also fails.
```
clang-macos> + cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/tools/clang-macos -DCMAKE_C_COMPILER=/usr/bin/clang -DCMAKE_CXX_COMPILER=/usr/bin/clang++ -DCMAKE_ASM_COMPILER=/usr/bin/clang -DLLVM_ENABLE_LIBCXX=ON -DLLVM_OPTIMIZED_TABLEGEN=ON -DLLVM_TARGETS_TO_BUILD=X86 -DLLVM_LINK_LLVM_DYLIB=ON ../../llvm
clang-macos> /build/build-clang-macos.sh: line 56: /build/CMake.app/Contents/bin/cmake: cannot execute binary file
Traceback (most recent call last):
File "/Users/jslaman/git/policy_sentry_bin/python-build-standalone/cpython-unix/build.py", line 906, in
sys.exit(main())
File "/Users/jslaman/git/policy_sentry_bin/python-build-standalone/cpython-unix/build.py", line 755, in main
client, get_image(client, ROOT, BUILD, "clang"), platform=platform
File "/Users/jslaman/git/policy_sentry_bin/python-build-standalone/cpython-unix/build.py", line 215, in build_clang
build_env.run(build_sh, environment=env)
File "/Users/jslaman/git/policy_sentry_bin/python-build-standalone/pythonbuild/buildenv.py", line 64, in run
container_exec(self.container, program, user=user, environment=environment)
File "/Users/jslaman/git/policy_sentry_bin/python-build-standalone/pythonbuild/docker.py", line 128, in container_exec
raise Exception("exit code %d from %s" % (inspect_res["ExitCode"], command))
Exception: exit code 126 from /build/build-clang-macos.sh
make: *** [/Users/jslaman/git/policy_sentry_bin/python-build-standalone/build/clang-8.0.1-macos.tar] Error 1
```
Not sure what I'm doing wrong..
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the failure with the provided Docker setup and pyoxidizer.bzl, starting at the dist target and pyoxidizer.bzl line 75 where to_python_executable() is called. Compare the URL and local Python distribution cases, then verify the macOS case separately; done means identifying the cause of the missing-file and exec-format errors and making the build produce a clear, actionable result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, python, rust
- Domain
- build-system, devtools
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100