JuliaPy / JuliaPy/PyCall.jl

`] build PyCall` does not work on NixOS

Đang mở
#952 10 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Julia
Star
1.5k
Fork
186
Chỉ số merge pull request
Không có pull request nào được merge trong 30 ngày

Mô tả

I'm running into some unexpected hiccups building PyCall.jl on NixOS. I have successfully used PyCall.jl on NixOS in the past, so I'm a bit stumped as to why it's borked now. I get the following failure:
```
(@v1.7) pkg> build PyCall
Building Conda ─→ `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/6cdc8832ba11c7695f494c9d9a1c31e90959ce0f/build.log`
Building PyCall → `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/71fd4022ecd0c6d20180e23ff1b3e05a143959c2/build.log`
ERROR: Error building `PyCall`:
PREFIX=/home/skainswo/.julia/conda/3
Unpacking payload ...
cannot execute /home/skainswo/.julia/conda/3/conda.exe: NIX_LD or NIX_LD_x86_64-linux is not set
cannot execute /home/skainswo/.julia/conda/3/conda.exe: NIX_LD or NIX_LD_x86_64-linux is not set
┌ Info: No system-wide Python was found; got the following error:
│ Base.IOError("could not spawn setenv(`/nix/store/nki9ywqzbvz68vr75kn2r7g1q84f5agy-python3-3.9.6/bin/python3 -c \"import distutils.sysconfig; print(distutils.sysconfig.get_config_var('VERSION'))\"`,[\"CXX=g++\", \"PATH=/nix/store/79zvzq15mga6bgk5phnmw0fd2pnpim9v-bash-interactive-5.1-p8/bin:/nix/store/f1n2zyq47sjn0qfhxvrdawyzlgk4sgax-patchelf-0.13/bin:/nix/store/7wn7xx8faqirzkvh5iwj26b024jzg09p-gcc-wrapper-10.3.0/bin:/nix/store/v819nrv8d33ns36gm4v9vqydq1v95axi-gcc-10.3.0/bin:/nix/store/dhj23pp1sld19shjslkzw3y9pr3qpi61-glibc-2.33-59-bin/bin:/nix/store/abc8hz0v47jzsf048j1qx7m94bxfhpl7-coreutils-9.0/bin:/nix/store/gxd0gvw88wyl9bqjc03ypm5zs7qbaf74-binutils-wrapper-2.35.2/bin:/nix/store/8d30avp4xj1w8krq7zax1mlxgs28663y-binutils-2.35.2/bin:/nix/store/qmlbjq46nhn1v08nnb23y2myrvra8hm7-julia-bin-1.7.1/bin:/nix/store/rppr9s436950i1dlzknbmz40m2xqqnxc-python3-3.9.9/bin:/nix/store/abc8hz0v47jzsf048j1qx7m94bxfhpl7-coreutils-9.0/bin:/nix/store/r87rxnys39173q5nrlqq46vpa2203rmk-findutils-4.8.0/bin:/nix/store/xgqrjhz9sx8bllvayz18sql2dfd89q4g-diffutils-3.8/bin:/nix/store/aw3vgxn4fdrjqgmc94fr582hjhr2z3p3-gnused-4.8/bin:/nix/store/rvdb7057bmg3rasbqq8lzl10dffy3p31-gnugrep-3.7/bin:/nix/store/3qmgmnz3msrk0cf0qgcyja0wzyfhpxc2-gawk-5.1.1/bin:/nix/store/xc1dbg8mc8hd6bq7zizxf5nyrljk8az0-gnutar-1.34/bin:/nix/store/1wads5y31dky89hxnn5zmiri3c72kdpc-gzip-1.11/bin:/nix/store/l8d7zz3ninmgg71738q485anfjh5yd6q-bzip2-1.0.6.0.2-bin/bin:/nix/store/ba439awj33pw4l4bj95k4dy58nx69b8v-gnumake-4.3/bin:/nix/store/iqprjr5k5385bhf1dzj07zwd5p43py1n-bash-5.1-p12/bin:/nix/store/22avws4xw0chqniqym3q0w4nhdcr0hh2-patch-2.7.6/bin:/nix/store/aq0v28n1ka8m1ayg6wrpbjy00bb7fj3k-xz-5.2.5-bin/bin\", \"HOST_PATH=/nix/store/qmlbjq46nhn1v08nnb23y2myrvra8hm7-julia-bin-1.7.1/bin:/nix/store/rppr9s436950i1dlzknbmz40m2xqqnxc-python3-3.9.9/bin:/nix/store/abc8hz0v47jzsf048j1qx7m94bxfhpl7-coreutils-9.0/bin:/nix/store/r87rxnys39173q5nrlqq46vpa2203rmk-findutils-4.8.0/bin:/nix/store/xgqrjhz9sx8bllvayz18sql2dfd89q4g-diffutils-3.8/bin:/nix/store/aw3vgxn4fdrjqgmc94fr582hjhr2z3p3-gnused-4.8/bin:/nix/store/rvdb7057bmg3rasbqq8lzl10dffy3p31-gnugrep-3.7/bin:/nix/store/3qmgmnz3msrk0cf0qgcyja0wzyfhpxc2-gawk-5.1.1/bin:/nix/store/xc1dbg8mc8hd6bq7zizxf5nyrljk8az0-gnutar-1.34/bin:/nix/store/1wads5y31dky89hxnn5zmiri3c72kdpc-gzip-1.11/bin:/nix/store/l8d7zz3ninmgg71738q485anfjh5yd6q-bzip2-1.0.6.0.2-bin/bin:/nix/store/ba439awj33pw4l4bj95k4dy58nx69b8v-gnumake-4.3/bin:/nix/store/iqprjr5k5385bhf1dzj07zwd5p43py1n-bash-5.1-p12/bin:/nix/store/22avws4xw0chqniqym3q0w4nhdcr0hh2-patch-2.7.6/bin:/nix/store/aq0v28n1ka8m1ayg6wrpbjy00bb7fj3k-xz-5.2.5-bin/bin\", \"doInstallCheck=\", \"CC=gcc\", \"_PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata__linux_x86_64-linux-gnu\", \"outputs=out\", \"TMP=/run/user/1000\", \"OBJCOPY=objcopy\", \"nobuildPhase=echo\\necho \\\"This derivation is not meant to be built, aborting\\\";\\necho\\nexit 1\\n\", \"SSL_CERT_FILE=/nix/store/ggqkxkqjn8hf7h1kgv5qz8lwas9i5z6k-nss-cacert-3.71/etc/ssl/certs/ca-bundle.crt\", \"USER=skainswo\", \"NIX_CC_WRAPPER_TARGET_HOST_x86_64_unknown_linux_gnu=1\", \"SYSTEM_CERTIFICATE_PATH=/nix/store/ggqkxkqjn8hf7h1kgv5qz8lwas9i5z6k-nss-cacert-3.71/etc/ssl/certs/ca-bundle.crt\", \"depsBuildTargetPropagated=\", \"LD=ld\", \"PYTHONHASHSEED=0\", \"NIX_BINTOOLS_WRAPPER_TARGET_HOST_x86_64_unknown_linux_gnu=1\", \"IN_NIX_SHELL=pure\", \"NM=nm\", \"STRIP=strip\", \"OBJDUMP=objdump\", \"XDG_DATA_DIRS=/nix/store/f1n2zyq47sjn0qfhxvrdawyzlgk4sgax-patchelf-0.13/share\", \"nativeBuildInputs=\", \"SHELL=/nix/store/79zvzq15mga6bgk5phnmw0fd2pnpim9v-bash-interactive-5.1-p8/bin/bash\", \"system=x86_64-linux\", \"depsBuildBuild=\", \"name=nix-shell\", \"RANLIB=ranlib\", \"TMPDIR=/run/user/1000\", \"NIX_SSL_CERT_FILE=/nix/store/ggqkxkqjn8hf7h1kgv5qz8lwas9i5z6k-nss-cacert-3.71/etc/ssl/certs/ca-bundle.crt\", \"buildInputs=/nix/store/ggqkxkqjn8hf7h1kgv5qz8lwas9i5z6k-nss-cacert-3.71 /nix/store/qmlbjq46nhn1v08nnb23y2myrvra8hm7-julia-bin-1.7.1 /nix/store/rppr9s436950i1dlzknbmz40m2xqqnxc-python3-3.9.9\", \"NIX_BUILD_TOP=/run/user/1000\", \"depsHostHostPropagated=\", \"depsTargetTargetPropagated=\", \"doCheck=\", \"depsBuildTarget=\", \"SOURCE_DATE_EPOCH=315532800\", \"STRINGS=strings\", \"HOME=/home/skainswo\", \"TERM=xterm-256color\", \"TEMP=/run/user/1000\", \"PYTHONIOENCODING=UTF-8\", \"PYTHONNOUSERSITE=1\", \"_PYTHON_HOST_PLATFORM=linux-x86_64\", \"NIX_CFLAGS_COMPILE= -frandom-seed=7q7jlhbyfg -isystem /nix/store/qmlbjq46nhn1v08nnb23y2myrvra8hm7-julia-bin-1.7.1/include -isystem /nix/store/rppr9s436950i1dlzknbmz40m2xqqnxc-python3-3.9.9/include -isystem /nix/store/qmlbjq46nhn1v08nnb23y2myrvra8hm7-julia-bin-1.7.1/include -isystem /nix/store/rppr9s436950i1dlzknbmz40m2xqqnxc-python3-3.9.9/include\", \"DETERMINISTIC_BUILD=1\", \"patches=\", \"READELF=readelf\", \"AS=as\", \"__ETC_PROFILE_SOURCED=1\", \"depsHostHost=\", \"TEMPDIR=/run/user/1000\", \"NIX_INDENT_MAKE=1\", \"strictDeps=\", \"shell=/nix/store/iqprjr5k5385bhf1dzj07zwd5p43py1n-bash-5.1-p12/bin/bash\", \"NIX_BINTOOLS=/nix/store/gxd0gvw88wyl9bqjc03ypm5zs7qbaf74-binutils-wrapper-2.35.2\", \"SIZE=size\", \"LOGNAME=skainswo\", \"SHLVL=2\", \"NIX_CC=/nix/store/7wn7xx8faqirzkvh5iwj26b024jzg09p-gcc-wrapper-10.3.0\", \"_=/nix/store/qmlbjq46nhn1v08nnb23y2myrvra8hm7-julia-bin-1.7.1/bin/julia\", \"PAGER=less\", \"configureFlags=\", \"depsTargetTarget=\", \"AR=ar\", \"NIX_ENFORCE_NO_NATIVE=1\", \"PWD=/home/skainswo/dev/research/julia/permuted_sgd\", \"NIX_STORE=/nix/store\", \"NIX_BUILD_CORES=2\", \"PYTHONPATH=/nix/store/rppr9s436950i1dlzknbmz40m2xqqnxc-python3-3.9.9/lib/python3.9/site-packages\", \"out=/nix/store/7q7jlhbyfgim8i3w055lihzwxqlidxh6-nix-shell\", \"builder=/nix/store/iqprjr5k5385bhf1dzj07zwd5p43py1n-bash-5.1-p12/bin/bash\", \"propagatedBuildInputs=\", \"depsBuildBuildPropagated=\", \"shellHook=\", \"phases=nobuildPhase\", \"CONFIG_SHELL=/nix/store/iqprjr5k5385bhf1dzj07zwd5p43py1n-bash-5.1-p12/bin/bash\", \"NIX_LDFLAGS=-rpath /nix/store/7q7jlhbyfgim8i3w055lihzwxqlidxh6-nix-shell/lib64 -rpath /nix/store/7q7jlhbyfgim8i3w055lihzwxqlidxh6-nix-shell/lib -L/nix/store/qmlbjq46nhn1v08nnb23y2myrvra8hm7-julia-bin-1.7.1/lib -L/nix/store/rppr9s436950i1dlzknbmz40m2xqqnxc-python3-3.9.9/lib -L/nix/store/qmlbjq46nhn1v08nnb23y2myrvra8hm7-julia-bin-1.7.1/lib -L/nix/store/rppr9s436950i1dlzknbmz40m2xqqnxc-python3-3.9.9/lib\", \"NIX_HARDENING_ENABLE=fortify stackprotector pic strictoverflow format relro bindnow\", \"JULIA_LOAD_PATH=@:/run/user/1000/jl_h44YES\", \"propagatedNativeBuildInputs=\", \"stdenv=/nix/store/lwcfw5vyjlzrs35k1hanv21j1q2s5c5w-stdenv-linux\", \"LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:\", \"OPENBLAS_MAIN_FREE=1\"]): no such file or directory (ENOENT)", -2)
└ using the Python distribution in the Conda package
[ Info: Downloading miniconda installer ...
[ Info: Installing miniconda ...
ERROR: LoadError: failed process: Process(`/home/skainswo/.julia/conda/3/installer.sh -b -f -p /home/skainswo/.julia/conda/3`, ProcessExited(1)) [1]

Stacktrace:
[1] pipeline_error
@ ./process.jl:531 [inlined]
[2] run(::Cmd; wait::Bool)
@ Base ./process.jl:446
[3] run
@ ./process.jl:444 [inlined]
[4] _install_conda(env::String, force::Bool)
@ Conda ~/.julia/packages/Conda/1403Y/src/Conda.jl:207
[5] _install_conda(env::String)
@ Conda ~/.julia/packages/Conda/1403Y/src/Conda.jl:193
[6] runconda(args::Cmd, env::String)
@ Conda ~/.julia/packages/Conda/1403Y/src/Conda.jl:128
[7] add(pkg::String, env::String; channel::String)
@ Conda ~/.julia/packages/Conda/1403Y/src/Conda.jl:223
[8] add (repeats 2 times)
@ ~/.julia/packages/Conda/1403Y/src/Conda.jl:222 [inlined]
[9] top-level scope
@ ~/.julia/packages/PyCall/L0fLP/deps/build.jl:79
[10] include(fname::String)
@ Base.MainInclude ./client.jl:451
[11] top-level scope
@ none:5
in expression starting at /home/skainswo/.julia/packages/PyCall/L0fLP/deps/build.jl:43
```
It appears to be hung up on some aspect of the conda installation. However, building Conda.jl works without any issues:
```
(@v1.7) pkg> build Conda
Building Conda → `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/6cdc8832ba11c7695f494c9d9a1c31e90959ce0f/build.log`
(@v1.7) pkg>
```
A few things are weird to me about this situation:
* Why is PyCall.jl failing to build due to issues with Conda while building Conda.jl works just fine?
* `cannot execute /home/skainswo/.julia/conda/3/conda.exe: NIX_LD or NIX_LD_x86_64-linux is not set` suggests to me that Conda.jl is smart enough to use [nix-ld](https://github.com/Mic92/nix-ld) when it's available, but I haven't been able to find anything about this in the docs or in the sources for Conda.jl or PyCall.jl. Where is this warning coming from?
* The error suggests that it's trying and failing to run `/nix/store/nki9ywqzbvz68vr75kn2r7g1q84f5agy-python3-3.9.6/bin/python3 -c \"import distutils.sysconfig; print(distutils.sysconfig.get_config_var('VERSION'))\"`, but this is very odd since I don't even have that version of python installed! I have python 3.9.9 with a different hash entirely! How is it hallucinating this python install that does not exist?

The main thing that has changed since the last time this worked for me is that I upgraded NixOS from 21.05 to 21.11.

Here's my shell.nix for reference:
```nix
let
# Last updated: 1/3/22. From status.nixos.org.
pkgs = import (fetchTarball ("https://github.com/NixOS/nixpkgs/archive/adf7f03d3bfceaba64788e1e846191025283b60d.tar.gz")) { };
in
pkgs.mkShell {
buildInputs = with pkgs; [
# See https://github.com/NixOS/nixpkgs/issues/66716. Necessary for julia to
# be able to download packages.
cacert

julia_17-bin

# Necessary for PyPlot.jl
python3
];
}
```
and here's the version of python that's actually available:
```
[nix-shell:~/dev/research/julia/permuted_sgd]$ which python
/nix/store/rppr9s436950i1dlzknbmz40m2xqqnxc-python3-3.9.9/bin/python
```
```
julia> versioninfo()
Julia Version 1.7.1
Commit ac5cc99908 (2021-12-22 19:35 UTC)
Platform Info:
OS: Linux (x86_64-pc-linux-gnu)
CPU: Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-12.0.1 (ORCJIT, skylake-avx512)
```

Hướng dẫn đóng góp

Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này

Hướng nghiên cứu

Tái hiện `pkg> build PyCall` trong `shell.nix` NixOS được cung cấp, sau đó kiểm tra `deps/build.jl` trong PyCall và các entry point `Conda.jl` được tham chiếu quanh `_install_conda` và `runconda`. Theo dõi tiến trình `conda.exe` và tiến trình Python bị lỗi trước khi kiểm tra việc nâng cấp NixOS ảnh hưởng đến chúng như thế nào. Hoàn tất khi PyCall build thành công trong môi trường này và đã hiểu hành vi Conda được báo cáo.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Đánh giá

Công nghệ
julia, nixos, python
Lĩnh vực
build-system, operating-systems
Loại issue
Lỗi
Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức độ hoạt động
Đình trệ
Độ rõ ràng
Khá rõ ràng
Mức phù hợp với người mới
35/100

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.