indygreg / indygreg/PyOxidizer

Pyoxidizier could not make python code with "import torch".

Open
#119 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
6.2k
Forks
256
PR merge metrics
No merged PRs in 30d

Description

(1)Environment
Ubuntu16.04
Pyoxidizer Ver0.2.0 & ver0.3.0
CUDA10.0
cuDnn7.1
Docker version.19.03.1

(2)First code
I tried to make Executable file by Pyoxidizer on docker container.
At first , I user Python code "import requests" as below.

・mod.py
import requests
r = requests.get('https://httpbin.org/get')
print(r.status_code)

・pyoxidizer.toml

### # This file controls the PyOxidizer build configuration. See the
### # pyoxidizer crate's documentation for extensive documentation
### # on this file format.

[[build]]
application_name = "pyapp"

[[embedded_python_config]]
raw_allocator = "jemalloc"
sys_paths = ["$ORIGIN/lib"]

#### dont_write_bytecode = true
### # ignore_environment = true
### # no_site = true
### # no_user_site_directory = true
### # optimize_level = 0
### # stdio_encoding = "utf-8:strict"
### # unbuffered_stdio = false
### #write_modules_directory_env = "PYOXIDIZER_WRITE_MODULES_DIR"

### # Windows doesn't support jemalloc.
[[embedded_python_config]]
build_target = "x86_64-pc-windows-msvc"
raw_allocator = "system"

[[packaging_rule]]
type = "stdlib-extensions-policy"

### # Package all available extension modules from the Python distribution.
### # The Python interpreter will be fully featured.
policy = "all"

### # Only package the minimal set of extension modules needed to initialize
### # a Python interpreter. Many common packages in Python's standard
### # library won't work with this setting.
### # policy = "minimal"

### # Only package extension modules that don't require linking against
### # non-Python libraries. e.g. will exclude support for OpenSSL, SQLite3,
### # other features that require external libraries.
### # policy = "no-libraries"

### # Explicit list of extension modules from the distribution to include.
### # [[packaging_rule]]
### # type = "stdlib-extensions-explicit-includes"
### # includes = ["binascii", "errno", "itertools", "math", "select", "_socket"]

### # Explicit list of extension modules from the distribution to exclude.
[[packaging_rule]]
type = "pip-install-simple"
package = "requests"
install_location = "app-relative:lib"
extra_args = ["--proxy=url:port and other args"]
### # excludes = ["_ssl"]

### # Package the entire Python standard library without sources.
[[packaging_rule]]
type = "stdlib"
include_source = false

### # Package .py files discovered in a local directory.
[[packaging_rule]]
type = "package-root"
path = "."
packages = ["mod"]

### # Package things from a populated virtualenv.
### # [[packaging_rule]]
### # type = "virtualenv"
### # path = "/path/to/venv"

### # Filter all resources collected so far through a filter of names
### # in a file.
### # [[packaging_rule]]
### # type = "filter-include"
### # files = ["/path/to/filter-file"]

### # How Python should run by default. This is only needed if you
### # call ``run()``. For applications customizing how the embedded
### # Python interpreter is invoked, this section is not relevant.
[[embedded_python_run]]
### # Run an interactive Python interpreter.
### #mode = "repl"
mode = "module"
module = "mod"
### # Import a Python module and run it.
### #mode = "module"
### #module = "mypackage.__main__"

### # Evaluate some Python code.
### #mode = "eval"
### #code = "from mypackage import main; main()"

### # END OF COMMON USER-ADJUSTED SETTINGS.
### #
### # Everything below this is typically managed by PyOxidizer and doesn't need
### # to be updated by people.

[[python_distribution]]
build_target = "x86_64-apple-darwin"
url = "https://github.com/indygreg/python-build-standalone/releases/download/20190617/cpython-3.7.3-macos-2019$
sha256 = "6668202a3225892ce252eff4bb53a58ac058b6a413ab9d37c026a500c2a561ee"
[[python_distribution]]
build_target = "x86_64-pc-windows-msvc"
url = "https://github.com/indygreg/python-build-standalone/releases/download/20190617/cpython-3.7.3-windows-am$
sha256 = "fd43554b5654a914846cf1c251d1ad366f46c7c4d20b7c44572251b533351221"
[[python_distribution]]
build_target = "x86_64-unknown-linux-gnu"
url = "https://github.com/indygreg/python-build-standalone/releases/download/20190617/cpython-3.7.3-linux64-20$
sha256 = "d6b80a9723c124d6d193f8816fdb874ba6d56abfb35cbfcc2b27de53176d0620"
[[python_distribution]]
build_target = "x86_64-unknown-linux-musl"
url = "https://github.com/indygreg/python-build-standalone/releases/download/20190617/cpython-3.7.3-linux64-mu$
sha256 = "2be2d109b82634b36685b89800887501b619ef946dda182e5a8ab5c7029a8136"
[[pyoxidizer]]
version = "0.2.0"
commit = ""

・Result
I could make executable file "pyapp" and it operated correctly

root@c9c062f80669:~/pyoxidizer/pyapp/build/apps/pyapp/debug# ls
LICENSE.bdb.txt LICENSE.libffi.txt LICENSE.ncurses.txt LICENSE.sqlite.txt pyapp
LICENSE.bzip2.txt LICENSE.liblzma.txt LICENSE.openssl.txt LICENSE.zlib.txt
LICENSE.libedit.txt LICENSE.libuuid.txt LICENSE.python.txt lib

・Executable file operated correctly
root@c9c062f80669:~/pyoxidizer/pyapp/build/apps/pyapp/debug# ./pyapp
200

(3) I tried to make executable file with "import torch" as below
torch1.py

### # -*- coding: utf-8 -*-
import torch

dtype = torch.float

device = torch.device("cpu")

device = torch.device("cuda:0") # Uncomment this to run on GPU

### # N is batch size; D_in is input dimension;

### # H is hidden dimension; D_out is output dimension.

N, D_in, H, D_out = 64, 1000, 100, 10

### # Create random input and output data

x = torch.randn(N, D_in, device=device, dtype=dtype)

y = torch.randn(N, D_out, device=device, dtype=dtype)

### # Randomly initialize weights

w1 = torch.randn(D_in, H, device=device, dtype=dtype)

w2 = torch.randn(H, D_out, device=device, dtype=dtype)

learning_rate = 1e-6

for t in range(500):

### # Forward pass: compute predicted y

h = x.mm(w1)

h_relu = h.clamp(min=0)

y_pred = h_relu.mm(w2)

# Compute and print loss

loss = (y_pred - y).pow(2).sum().item()

print(t, loss)

### # Backprop to compute gradients of w1 and w2 with respect to loss

grad_y_pred = 2.0 * (y_pred - y)

grad_w2 = h_relu.t().mm(grad_y_pred)
grad_h_relu = grad_y_pred.mm(w2.t())

grad_h = grad_h_relu.clone()

grad_h[h < 0] = 0

grad_w1 = x.t().mm(grad_h)

### # Update weights using gradient descent
w1 -= learning_rate * grad_w1

w2 -= learning_rate * grad_w2

(4) I checked that torch1.py operated correctly by python

root@ea287fee23c9:~/pyoxidizer/pyapp# python torch1.py
0 34049896.0
1 35127840.0
2 41991524.0
3 46098128.0
4 39011568.0
5 23425828.0
6 10408400.0
7 4270037.0
8 2077392.375
9 1302221.75
10 969095.6875
11 781831.375
12 652404.625
13 552934.75
14 472515.90625
15 406305.125
16 351212.96875
17 304976.9375
18 265925.375
19 232764.53125
20 204419.28125
21 180092.921875
22 159147.09375
23 141008.65625
24 125251.734375
25 111528.9765625
26 99531.3203125
27 89008.953125
28 79753.8671875

479 5.8306883147452027e-05
480 5.7573095546104014e-05
481 5.681748007191345e-05
482 5.608148785540834e-05
483 5.531605347641744e-05
484 5.4345680837286636e-05
485 5.3692849178332835e-05
486 5.318575495039113e-05
487 5.266119114821777e-05
488 5.196596612222493e-05
489 5.128789780428633e-05
490 5.061098636360839e-05
491 5.011096800444648e-05
492 4.9439848226029426e-05
493 4.890610580332577e-05
494 4.817677108803764e-05
495 4.762996832141653e-05
496 4.719095159089193e-05
497 4.6559551265090704e-05
498 4.621376137947664e-05
499 4.5695014705415815e-05

・pyoxidizer.toml
### # This file controls the PyOxidizer build configuration. See the
### # pyoxidizer crate's documentation for extensive documentation
### # on this file format.

[[build]]
application_name = "pyapp"

[[embedded_python_config]]
raw_allocator = "jemalloc"
sys_paths = ["$ORIGIN/lib"]

### # dont_write_bytecode = true
### # ignore_environment = true
### # no_site = true
### # no_user_site_directory = true
### # optimize_level = 0
### # stdio_encoding = "utf-8:strict"
### # unbuffered_stdio = false
### #write_modules_directory_env = "PYOXIDIZER_WRITE_MODULES_DIR"

### # Windows doesn't support jemalloc.
[[embedded_python_config]]
build_target = "i686-pc-windows-msvc"
raw_allocator = "system"

[[embedded_python_config]]
build_target = "x86_64-pc-windows-msvc"
raw_allocator = "system"

[[packaging_rule]]
type = "stdlib-extensions-policy"

### # Package all available extension modules from the Python distribution.
### # The Python interpreter will be fully featured.
policy = "all"

### # Only package the minimal set of extension modules needed to initialize
### # a Python interpreter. Many common packages in Python's standard
### # library won't work with this setting.
### # policy = "minimal"

### # Only package extension modules that don't require linking against
### # non-Python libraries. e.g. will exclude support for OpenSSL, SQLite3,
### # other features that require external libraries.
### # policy = "no-libraries"

### # Explicit list of extension modules from the distribution to include.
### # [[packaging_rule]]
### # type = "stdlib-extensions-explicit-includes"
### # includes = ["binascii", "errno", "itertools", "math", "select", "_socket"]

### # Explicit list of extension modules from the distribution to exclude.
[[packaging_rule]]
type = "pip-install-simple"
package = "torch"
### #extra_args = ["--proxy=url:port and other args"]
### # excludes = ["_ssl"]
### # Package the entire Python standard library without sources.
[[packaging_rule]]
type = "stdlib"
include_source = false

### # Write out license files next to the produced binary.
[[packaging_rule]]
type = "write-license-files"
path = ""

### # Package .py files discovered in a local directory.
[[packaging_rule]]
type = "package-root"
path = "."
packages = ["torch1"]

### # Package things from a populated virtualenv.
### # [[packaging_rule]]
### # type = "virtualenv"
### # path = "/path/to/venv"

### # Filter all resources collected so far through a filter of names
### # in a file.
### # [[packaging_rule]]
### # type = "filter-include"
### # files = ["/path/to/filter-file"]

### # How Python should run by default. This is only needed if you
### # call ``run()``. For applications customizing how the embedded
### # Python interpreter is invoked, this section is not relevant.
[[embedded_python_run]]
### # Run an interactive Python interpreter.
### #mode = "repl"
### mode = "module"
### module = "torch1"

### # Import a Python module and run it.
### #mode = "module"
### #module = "mypackage.__main__"

### # Evaluate some Python code.
### #mode = "eval"
### #code = "from mypackage import main; main()"

[[distribution]]
type = "wix"

### # END OF COMMON USER-ADJUSTED SETTINGS.
### #
### # Everything below this is typically managed by PyOxidizer and doesn't need
### # to be updated by people.

[[python_distribution]]
build_target = "i686-pc-windows-msvc"
url = "https://github.com/indygreg/python-build-standalone/releases/download/20190816/cpython-3.7.4-windows-x8$
sha256 = "46c77de57c5ebbcc8c25e1003a77ca827763da01455b097f4e5fc0b782526c9b"
[[python_distribution]]
build_target = "x86_64-apple-darwin"
url = "https://github.com/indygreg/python-build-standalone/releases/download/20190816/cpython-3.7.4-macos-2019$
sha256 = "4a77d5ca898196bbc977eb126d129340bab14fb6a8feaaa335675613852071de"
[[python_distribution]]
build_target = "x86_64-pc-windows-msvc"
url = "https://github.com/indygreg/python-build-standalone/releases/download/20190816/cpython-3.7.4-windows-am$
sha256 = "82ae15f31178c9854bacb5d59e00305c6f6080649c9960a29be6b92517b8e5e5"
[[pyoxidizer]]
version = "0.3.0"
commit = ""

・Compile Error
root@ea287fee23c9:~/pyoxidizer/pyapp# pyoxidizer build
no existing PyOxidizer artifacts found
processing config file /root/pyoxidizer/pyapp/pyoxidizer.toml
resolving Python distribution...
Python distribution available at /root/pyoxidizer/pyapp/build/target/x86_64-unknown-linux-gnu/debug/pyoxidizer/cpython-3.7.4-linux64-20190817T0224.tar.zst
reading data from Python distribution...
distribution info: PythonDistributionMinimalInfo {
flavor: "cpython",
version: "3.7.4",
os: "linux",
arch: "x86_64",
extension_modules: [
"_abc",
"_ast",
"_asyncio",
"_bisect",
"_blake2",
"_bz2",
"_codecs",
"_codecs_cn",
"_codecs_hk",
"_codecs_iso2022",
"_codecs_jp",
"_codecs_kr",
"_codecs_tw",
"_collections",
"_contextvars",
"_crypt",
"_csv",
"_ctypes",
"_ctypes_test",
"_curses",
"_curses_panel",
"_datetime",
"_dbm",
"_decimal",
"_elementtree",
"_functools",
"_hashlib",
"_heapq",
"_imp",
"_io",
"_json",
"_locale",
"_lsprof",
"_lzma",

adding embedded extension module: termios
adding embedded extension module: time
adding embedded extension module: unicodedata
adding embedded extension module: xxlimited
adding embedded extension module: xxsubtype
adding embedded extension module: zipimport
adding embedded extension module: zlib
processing packaging rule: PipInstallSimple(PackagingPipInstallSimple { package: "torch", optimize_level: 0, excludes: [], include_source: true, install_location: Embedded, extra_args: None })
installing modified distutils to /tmp/pyoxidizer-pip-install.J7yOOm46DLMv/packages
modifying distutils/_msvccompiler.py for oxidation
modifying distutils/command/build_ext.py for oxidation
modifying distutils/unixccompiler.py for oxidation
pip installing to /tmp/pyoxidizer-pip-install.J7yOOm46DLMv/install
Collecting torch
Using cached https://files.pythonhosted.org/packages/f8/02/880b468bd382dc79896eaecbeb8ce95e9c4b99a24902874a2cef0b562cea/torch-0.1.2.post2.tar.gz
ERROR: Command errored out with exit status 1:
command: /root/pyoxidizer/pyapp/build/target/x86_64-unknown-linux-gnu/debug/pyoxidizer/python.1d3b5dc07ee2/python/install/bin/python3.7m -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-6sgi4bfk/torch/setup.py'"'"'; __file__='"'"'/tmp/pip-install-6sgi4bfk/torch/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base pip-egg-info
cwd: /tmp/pip-install-6sgi4bfk/torch/
Complete output (8 lines):
running egg_info
creating pip-egg-info/torch.egg-info
writing pip-egg-info/torch.egg-info/PKG-INFO
writing dependency_links to pip-egg-info/torch.egg-info/dependency_links.txt
writing requirements to pip-egg-info/torch.egg-info/requires.txt
writing top-level names to pip-egg-info/torch.egg-info/top_level.txt
writing manifest file 'pip-egg-info/torch.egg-info/SOURCES.txt'
error: package directory 'torch/cuda' does not exist
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
thread 'main' panicked at 'error running pip', /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/pyoxidizer-0.3.0/src/pyrepackager/packaging_rule.rs:889:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

・torch1.py Result
I could not make executable error.
I only imported one module "torch" in this code.

I'd really appriciate it if anyone teach me the way to delete this error !
 

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the pyoxidizer.toml configuration and torch1.py, then reproduce the build in the stated Ubuntu Docker environment. The compile-error output is truncated in this issue, so capture the complete error and compare the torch packaging setup with the working requests example. Done means the generated pyapp builds and runs torch1.py successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
build-system, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.