JuliaPy / JuliaPy/PythonCall.jl

Precompilation after add fails for packages with python dependencies

Open
#266 10 comments 0 reactions 0 assignees View on GitHub
Dominant language
Julia
Stars
1.1k
Forks
86
Avg merge
1d 22h
Merged PRs (30d)
3

Description

Thanks alot for this package!

Example on windows below. Running `using Faiss` afterwards installs everything correctly so despite the alarming title it is more of an inconvenience.

Error becomes harder to reproduce after a few attempts so here I use a clean julia depot (using `$Env:JULIA_DEPOT_PATH = "c:\temp\.juliatmp"` in powershell).

Could not reproduce on linux.

```julia
julia> versioninfo()
Julia Version 1.8.3
Commit 0434deb161 (2022-11-14 20:14 UTC)
Platform Info:
OS: Windows (x86_64-w64-mingw32)
CPU: 8 × 11th Gen Intel(R) Core(TM) i5-1145G7 @ 2.60GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-13.0.1 (ORCJIT, tigerlake)
Threads: 1 on 8 virtual cores
Environment:
JULIA_DEPOT_PATH = c:\temp\.juliatmp

(v1.8) pkg> add Faiss
Installing known registries into `c:\temp\.juliatmp`
Updating registry at `c:\temp\.juliatmp\registries\General.toml`
Resolving package versions...
Installed MicroMamba ────────────────── v0.1.12
Installed Parsers ───────────────────── v2.5.4
Installed TableTraits ───────────────── v1.0.1
Installed Tables ────────────────────── v1.10.0
Installed Scratch ───────────────────── v1.1.1
Installed JSON3 ─────────────────────── v1.12.0
Installed Preferences ───────────────── v1.3.0
Installed PythonCall ────────────────── v0.9.10
Installed DataAPI ───────────────────── v1.14.0
Installed OrderedCollections ────────── v1.4.1
Installed micromamba_jll ────────────── v1.2.0+0
Installed IteratorInterfaceExtensions ─ v1.0.0
Installed DataValueInterfaces ───────── v1.0.0
Installed Pidfile ───────────────────── v1.3.0
Installed SnoopPrecompile ───────────── v1.0.3
Installed JLLWrappers ───────────────── v1.4.1
Installed Requires ──────────────────── v1.3.0
Installed Faiss ─────────────────────── v0.3.5
Installed UnsafePointers ────────────── v1.0.0
Installed MacroTools ────────────────── v0.5.10
Installed CondaPkg ──────────────────── v0.2.17
Installed StructTypes ───────────────── v1.10.0
Updating `C:\temp\.juliatmp\environments\v1.8\Project.toml`
[97d83715] + Faiss v0.3.5
Updating `C:\temp\.juliatmp\environments\v1.8\Manifest.toml`
[992eb4ea] + CondaPkg v0.2.17
[9a962f9c] + DataAPI v1.14.0
[e2d170a0] + DataValueInterfaces v1.0.0
[97d83715] + Faiss v0.3.5
[82899510] + IteratorInterfaceExtensions v1.0.0
[692b3bcd] + JLLWrappers v1.4.1
[0f8b85d8] + JSON3 v1.12.0
[1914dd2f] + MacroTools v0.5.10
[0b3b1443] + MicroMamba v0.1.12
[bac558e1] + OrderedCollections v1.4.1
[69de0a69] + Parsers v2.5.4
[fa939f87] + Pidfile v1.3.0
[21216c6a] + Preferences v1.3.0
[6099a3de] + PythonCall v0.9.10
[ae029012] + Requires v1.3.0
[6c6a2e73] + Scratch v1.1.1
[66db9d55] + SnoopPrecompile v1.0.3
[856f2bd8] + StructTypes v1.10.0
[3783bdb8] + TableTraits v1.0.1
[bd369af6] + Tables v1.10.0
[e17b2a0c] + UnsafePointers v1.0.0
[f8abcde7] + micromamba_jll v1.2.0+0
[0dad84c5] + ArgTools v1.1.1
[56f22d72] + Artifacts
[2a0f44e3] + Base64
[ade2ca70] + Dates
[f43a241f] + Downloads v1.6.0
[7b1f6079] + FileWatching
[b77e0a4c] + InteractiveUtils
[4af54fe1] + LazyArtifacts
[b27032c2] + LibCURL v0.6.3
[76f85450] + LibGit2
[8f399da3] + Libdl
[37e2e46d] + LinearAlgebra
[56ddb016] + Logging
[d6f4376e] + Markdown
[a63ad114] + Mmap
[ca575930] + NetworkOptions v1.2.0
[44cfe95a] + Pkg v1.8.0
[de0858da] + Printf
[3fa0cd96] + REPL
[9a3f8284] + Random
[ea8e919c] + SHA v0.7.0
[9e88b42a] + Serialization
[6462fe0b] + Sockets
[fa267f1f] + TOML v1.0.0
[a4e569a6] + Tar v1.10.1
[8dfed614] + Test
[cf7118a7] + UUIDs
[4ec0a83e] + Unicode
[e66e0078] + CompilerSupportLibraries_jll v0.5.2+0
[deac9b47] + LibCURL_jll v7.84.0+0
[29816b5a] + LibSSH2_jll v1.10.2+0
[c8ffd9c3] + MbedTLS_jll v2.28.0+0
[14a3606d] + MozillaCACerts_jll v2022.2.1
[4536629a] + OpenBLAS_jll v0.3.20+0
[83775a58] + Zlib_jll v1.2.12+3
[8e850b90] + libblastrampoline_jll v5.1.1+0
[8e850ede] + nghttp2_jll v1.48.0+0
[3f19e933] + p7zip_jll v17.4.0+0
Precompiling project...
✗ Faiss
27 dependencies successfully precompiled in 35 seconds
1 dependency errored. To see a full report either run `import Pkg; Pkg.precompile()` or load the package

(v1.8) pkg> precompile
Precompiling project...
✗ Faiss
0 dependencies successfully precompiled in 8 seconds. 27 already precompiled.

ERROR: The following 1 direct dependency failed to precompile:

Faiss [97d83715-92a6-43f6-a1c3-b293f6dd920a]

Failed to precompile Faiss [97d83715-92a6-43f6-a1c3-b293f6dd920a] to c:\temp\.juliatmp\compiled\v1.8\Faiss\jl_FF33.tmp.
CondaPkg Found dependencies: c:\temp\.juliatmp\packages\PythonCall\3GRYN\CondaPkg.toml
CondaPkg Found dependencies: c:\temp\.juliatmp\packages\Faiss\18CT2\CondaPkg.toml
CondaPkg Resolving changes
+ faiss-cpu
+ numpy
+ python
CondaPkg Creating environment
│ c:\temp\.juliatmp\artifacts\4b82d7ce236527959584cbf6b4cd3a10545b99ac\bin\micromamba.exe
│ -r c:\temp\.juliatmp\scratchspaces\0b3b1443-0f03-428d-bdfb-f27f9c1191ea\root
│ create
│ -y
│ -p c:\temp\.juliatmp\environments\v1.8\.CondaPkg\env
│ --override-channels
│ --no-channel-priority
│ faiss-cpu[version='>=1.7,<2']
│ numpy[version='>1.10']
│ python[version='>=3.7,<4',channel='conda-forge',build='*cpython*']
│ python[version='>=3.6,<4']
│ -c conda-forge
└ -c pytorch
ERROR: LoadError: InitError: failed process: Process(`'c:\temp\.juliatmp\artifacts\4b82d7ce236527959584cbf6b4cd3a10545b99ac\bin\micromamba.exe' -r 'c:\temp\.juliatmp\scratchspaces\0b3b1443-0f03-428d-bdfb-f27f9c1191ea\root' create -y -p 'c:\temp\.juliatmp\environments\v1.8\.CondaPkg\env' --override-channels --no-channel-priority "faiss-cpu[version='>=1.7,<2']" "numpy[version='>1.10']" "python[version='>=3.7,<4',channel='conda-forge',build='*cpython*']" "python[version='>=3.6,<4']" -c conda-forge -c pytorch`, ProcessExited(3221226505)) [3221226505]

Stacktrace:
[1] pipeline_error
@ .\process.jl:565 [inlined]
[2] run(::Cmd; wait::Bool)
@ Base .\process.jl:480
[3] run(::Cmd)
@ Base .\process.jl:477
[4] _run(io::IO, cmd::Cmd, args::Any; flags::Any)
@ CondaPkg c:\temp\.juliatmp\packages\CondaPkg\j2Cct\src\resolve.jl:370
[5] _resolve_conda_install(io::Any, conda_env::Any, specs::Any, channels::Any; create::Any)
@ CondaPkg c:\temp\.juliatmp\packages\CondaPkg\j2Cct\src\resolve.jl:271
[6] resolve(; force::Bool, io::IO, interactive::Bool, dry_run::Bool)
@ CondaPkg c:\temp\.juliatmp\packages\CondaPkg\j2Cct\src\resolve.jl:527
[7] resolve()
@ CondaPkg c:\temp\.juliatmp\packages\CondaPkg\j2Cct\src\resolve.jl:384
[8] envdir()
@ CondaPkg c:\temp\.juliatmp\packages\CondaPkg\j2Cct\src\env.jl:70
[9] init_context()
@ PythonCall.C c:\temp\.juliatmp\packages\PythonCall\3GRYN\src\cpython\context.jl:63
[10] __init__()
@ PythonCall.C c:\temp\.juliatmp\packages\PythonCall\3GRYN\src\cpython\CPython.jl:21
[11] _include_from_serialized(pkg::Base.PkgId, path::String, depmods::Vector{Any})
@ Base .\loading.jl:831
[12] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt64)
@ Base .\loading.jl:1039
[13] _require(pkg::Base.PkgId)
@ Base .\loading.jl:1315
[14] _require_prelocked(uuidkey::Base.PkgId)
@ Base .\loading.jl:1200
[15] macro expansion
@ .\loading.jl:1180 [inlined]
[16] macro expansion
@ .\lock.jl:223 [inlined]
[17] require(into::Module, mod::Symbol)
@ Base .\loading.jl:1144
[18] include
@ .\Base.jl:419 [inlined]
[19] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::Nothing)
@ Base .\loading.jl:1554
[20] top-level scope
@ stdin:1
during initialization of module C
in expression starting at c:\temp\.juliatmp\packages\Faiss\18CT2\src\Faiss.jl:1
in expression starting at stdin:1
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.