JuliaLang / JuliaLang/Distributed.jl

Loading/precompiling packages requires them to be available on Distributed workers

Open
#80 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Julia
Stars
55
Forks
19
No merged PRs in 30d

Description

I have the julia REPL running on my local machine and I use Distributed to connect to a headless remote server via ssh. I want to use these workers to do some @distributed computations and then plot the results on my local REPL, e.g. using GLMakie.

As GLMakie requires X11 and graphics drivers to be installed, it is only installed on my local machine but not on the server. When I try to include GLMakie or call it after having connected to the headless server, the call fails.

Example:

using GLMakie
using Distributed

addprocs([...]) # This should connect to the headless server

scatter(randn(10), randn(10))

This fails with the error message

Errors encountered while load FileIO.File{FileIO.DataFormat{:PNG}, String}("/home/Uroc327/.julia/packages/Makie/lgPZh/assets/icons/icon-128.png").
All errors:
===========================================
On worker 2:
ArgumentError: Package PNGFiles [f57f5aa1-a3ce-4bc8-8ab9-96f992907883] is required but does not seem to be installed:
 - Run `Pkg.instantiate()` to install all recorded dependencies.

Stacktrace:
 [1] _require
   @ ./loading.jl:1012
 [2] require
   @ ./loading.jl:936
 [3] JuliaLang/julia#1
   @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/Distributed.jl:79
 [4] JuliaLang/julia#103
   @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:274
 [5] run_work_thunk
   @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:63
 [6] run_work_thunk
   @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:72
 [7] JuliaLang/julia#96
   @ ./task.jl:411
===========================================
ArgumentError: Package ImageMagick [6218d12a-5da1-5696-b52f-db25d2ecc6d1] is required but does not seem to be installed:
 - Run `Pkg.instantiate()` to install all recorded dependencies.

===========================================

Fatal error:
Error showing value of type Makie.FigureAxisPlot:
ERROR: On worker 2:
ArgumentError: Package PNGFiles [f57f5aa1-a3ce-4bc8-8ab9-96f992907883] is required but does not seem to be installed:
 - Run `Pkg.instantiate()` to install all recorded dependencies.

Stacktrace:
 [1] _require
   @ ./loading.jl:1012
 [2] require
   @ ./loading.jl:936
 [3] JuliaLang/julia#1
   @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/Distributed.jl:79
 [4] JuliaLang/julia#103
   @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:274
 [5] run_work_thunk
   @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:63
 [6] run_work_thunk
   @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:72
 [7] JuliaLang/julia#96
   @ ./task.jl:411
Stacktrace:
  [1] macro expansion
    @ ./task.jl:388 [inlined]
  [2] _require_callback(mod::Base.PkgId)
    @ Distributed /builddir/julia-1.6.1/usr/share/julia/stdlib/v1.6/Distributed/src/Distributed.jl:76
  [3] JuliaLang/julia#1
    @ ~/.julia/packages/ImageIO/CzrzY/src/ImageIO.jl:54 [inlined]
  [4] lock(f::ImageIO.var"#1#2"{Base.PkgId}, l::ReentrantLock)
    @ Base ./lock.jl:187
  [5] checked_import
    @ ~/.julia/packages/ImageIO/CzrzY/src/ImageIO.jl:53 [inlined]
  [6] load(f::FileIO.File{FileIO.DataFormat{:PNG}, String}; kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ ImageIO ~/.julia/packages/ImageIO/CzrzY/src/ImageIO.jl:59
  [7] load(f::FileIO.File{FileIO.DataFormat{:PNG}, String})
    @ ImageIO ~/.julia/packages/ImageIO/CzrzY/src/ImageIO.jl:59
  [8] #invokelatest#2
    @ ./essentials.jl:708 [inlined]
  [9] invokelatest
    @ ./essentials.jl:706 [inlined]
 [10] action(::Symbol, ::Vector{Union{Base.PkgId, Module}}, ::FileIO.Formatted; options::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ FileIO ~/.julia/packages/FileIO/u9YLx/src/loadsave.jl:219
 [11] action
    @ ~/.julia/packages/FileIO/u9YLx/src/loadsave.jl:197 [inlined]
 [12] action(::Symbol, ::Vector{Union{Base.PkgId, Module}}, ::Symbol, ::String; options::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ FileIO ~/.julia/packages/FileIO/u9YLx/src/loadsave.jl:185
 [13] action
    @ ~/.julia/packages/FileIO/u9YLx/src/loadsave.jl:185 [inlined]
 [14] load(::String; options::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ FileIO ~/.julia/packages/FileIO/u9YLx/src/loadsave.jl:113
 [15] load
    @ ~/.julia/packages/FileIO/u9YLx/src/loadsave.jl:110 [inlined]
 [16] _broadcast_getindex_evalf
    @ ./broadcast.jl:648 [inlined]
 [17] _broadcast_getindex
    @ ./broadcast.jl:621 [inlined]
 [18] getindex
    @ ./broadcast.jl:575 [inlined]
 [19] copy
    @ ./broadcast.jl:922 [inlined]
 [20] materialize
    @ ./broadcast.jl:883 [inlined]
 [21] icon()
    @ Makie ~/.julia/packages/Makie/lgPZh/src/Makie.jl:262
 [22] GLMakie.Screen(; resolution::Tuple{Int64, Int64}, visible::Bool, title::String, kw_args::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ GLMakie ~/.julia/packages/GLMakie/TfSB1/src/screen.jl:349
 [23] Screen
    @ ~/.julia/packages/GLMakie/TfSB1/src/screen.jl:303 [inlined]
 [24] global_gl_screen
    @ ~/.julia/packages/GLMakie/TfSB1/src/screen.jl:249 [inlined]
 [25] global_gl_screen(resolution::Tuple{Int64, Int64}, visibility::Bool, tries::Int64)
    @ GLMakie ~/.julia/packages/GLMakie/TfSB1/src/screen.jl:394
 [26] global_gl_screen
    @ ~/.julia/packages/GLMakie/TfSB1/src/screen.jl:393 [inlined]
 [27] backend_display
    @ ~/.julia/packages/GLMakie/TfSB1/src/display.jl:2 [inlined]
 [28] display(scene::Scene)
    @ Makie ~/.julia/packages/Makie/lgPZh/src/display.jl:59
 [29] #display#915
    @ ~/.julia/packages/Makie/lgPZh/src/display.jl:53 [inlined]
 [30] display
    @ ~/.julia/packages/Makie/lgPZh/src/display.jl:53 [inlined]
 [31] #display#914
    @ ~/.julia/packages/Makie/lgPZh/src/display.jl:52 [inlined]
 [32] display(fap::Makie.FigureAxisPlot)
    @ Makie ~/.julia/packages/Makie/lgPZh/src/display.jl:52
 [33] (::REPL.var"#do_respond#61"{Bool, Bool, REPL.var"#72#82"{REPL.LineEditREPL, REPL.REPLHistoryProvider}, REPL.LineEditREPL, REPL.LineEdit.Prompt})(s::REPL.LineEdit.MIState, buf::Any, ok::Bool)
    @ REPL /builddir/julia-1.6.1/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:798
Stacktrace:
  [1] handle_error(e::CompositeException, q::Base.PkgId, bt::Vector{Union{Ptr{Nothing}, Base.InterpreterIP}})
    @ FileIO ~/.julia/packages/FileIO/u9YLx/src/error_handling.jl:61
  [2] handle_exceptions(exceptions::Vector{Tuple{Any, Union{Base.PkgId, Module}, Vector{T} where T}}, action::String)
    @ FileIO ~/.julia/packages/FileIO/u9YLx/src/error_handling.jl:56
  [3] action(::Symbol, ::Vector{Union{Base.PkgId, Module}}, ::FileIO.Formatted; options::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ FileIO ~/.julia/packages/FileIO/u9YLx/src/loadsave.jl:228
  [4] action
    @ ~/.julia/packages/FileIO/u9YLx/src/loadsave.jl:197 [inlined]
  [5] action(::Symbol, ::Vector{Union{Base.PkgId, Module}}, ::Symbol, ::String; options::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ FileIO ~/.julia/packages/FileIO/u9YLx/src/loadsave.jl:185
  [6] action
    @ ~/.julia/packages/FileIO/u9YLx/src/loadsave.jl:185 [inlined]
  [7] load(::String; options::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ FileIO ~/.julia/packages/FileIO/u9YLx/src/loadsave.jl:113
  [8] load
    @ ~/.julia/packages/FileIO/u9YLx/src/loadsave.jl:110 [inlined]
  [9] _broadcast_getindex_evalf
    @ ./broadcast.jl:648 [inlined]
 [10] _broadcast_getindex
    @ ./broadcast.jl:621 [inlined]
 [11] getindex
    @ ./broadcast.jl:575 [inlined]
 [12] copy
    @ ./broadcast.jl:922 [inlined]
 [13] materialize
    @ ./broadcast.jl:883 [inlined]
 [14] icon()
    @ Makie ~/.julia/packages/Makie/lgPZh/src/Makie.jl:262
 [15] GLMakie.Screen(; resolution::Tuple{Int64, Int64}, visible::Bool, title::String, kw_args::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ GLMakie ~/.julia/packages/GLMakie/TfSB1/src/screen.jl:349
 [16] Screen
    @ ~/.julia/packages/GLMakie/TfSB1/src/screen.jl:303 [inlined]
 [17] global_gl_screen
    @ ~/.julia/packages/GLMakie/TfSB1/src/screen.jl:249 [inlined]
 [18] global_gl_screen(resolution::Tuple{Int64, Int64}, visibility::Bool, tries::Int64)
    @ GLMakie ~/.julia/packages/GLMakie/TfSB1/src/screen.jl:394
 [19] global_gl_screen
    @ ~/.julia/packages/GLMakie/TfSB1/src/screen.jl:393 [inlined]
 [20] backend_display
    @ ~/.julia/packages/GLMakie/TfSB1/src/display.jl:2 [inlined]
 [21] display(scene::Scene)
    @ Makie ~/.julia/packages/Makie/lgPZh/src/display.jl:59
 [22] #display#915
    @ ~/.julia/packages/Makie/lgPZh/src/display.jl:53 [inlined]
 [23] display
    @ ~/.julia/packages/Makie/lgPZh/src/display.jl:53 [inlined]
 [24] #display#914
    @ ~/.julia/packages/Makie/lgPZh/src/display.jl:52 [inlined]
 [25] display(fap::Makie.FigureAxisPlot)
    @ Makie ~/.julia/packages/Makie/lgPZh/src/display.jl:52
 [26] (::REPL.var"#do_respond#61"{Bool, Bool, REPL.var"#72#82"{REPL.LineEditREPL, REPL.REPLHistoryProvider}, REPL.LineEditREPL, REPL.LineEdit.Prompt})(s::REPL.LineEdit.MIState, buf::Any, ok::Bool)
    @ REPL /builddir/julia-1.6.1/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:798

When including GLMakie after having connected to the remote machine,

[...]
addprocs([...])
using GLMakie

the error message is also of the type is required but does not seem to be installed.

julia> versioninfo()
Julia Version 1.6.1
Commit 6aaedecc44* (2021-04-23 05:59 UTC)
Platform Info:
  OS: Linux (x86_64-unknown-linux-gnu)
  CPU: Intel(R) Core(TM) i7-10510U CPU @ 1.80GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.0.1 (ORCJIT, skylake)

julia> ] st
      Status `~/.julia/environments/v1.6/Project.toml`
  [6e4b80f9] BenchmarkTools v1.3.1
  [1e616198] COSMO v0.8.5
  [13f3f980] CairoMakie v0.7.5
  [79afa230] Continuables v1.0.1
  [f65535da] Convex v0.15.1
  [f68482b8] Cthulhu v1.6.1
  [a93c6f00] DataFrames v1.3.2
  [31c24e10] Distributions v0.25.53
  [634d3b9d] DrWatson v2.9.1
  [2fe49d83] Expectations v1.8.0
  [7a1cc6ca] FFTW v1.4.6
  [41a02a25] Folds v0.2.7
  [e9467ef8] GLMakie v0.5.5
  [c3a54625] JET v0.4.6
  [033835bb] JLD2 v0.4.22
  [4076af6c] JuMP v0.23.2
  [b964fa9f] LaTeXStrings v1.3.0
  [b4c46c6c] LinuxPerf v0.3.2
  [2ab3a3ac] LogExpFunctions v0.3.10
  [bdcacae8] LoopVectorization v0.12.103
  [23992714] MAT v0.10.3
  [b8f27783] MathOptInterface v1.1.1
  [eff96d63] Measurements v2.7.1
  [01bcebdf] Nonconvex v1.0.2
  [5fb14364] OhMyREPL v0.5.12
  [429524aa] Optim v1.6.2
  [e4faabce] PProf v2.1.3
  [9b87118b] PackageCompiler v2.0.5
  [91a5bcdd] Plots v1.27.3
  [c3e4b0f8] Pluto v0.18.4
  [7f904dfe] PlutoUI v0.7.38
  [92933f4c] ProgressMeter v1.7.2
  [295af30f] Revise v3.2.1
  [f2b01f46] Roots v1.4.0
  [276daf66] SpecialFunctions v2.1.4
  [90137ffa] StaticArrays v1.4.3
  [2913bbd2] StatsBase v0.33.16
  [f3b207a7] StatsPlots v0.14.33
  [0c5d862f] Symbolics v4.3.0
  [a759f4b9] TimerOutputs v0.5.16
  [37b6cedf] Traceur v0.3.1
  [28d57a85] Transducers v0.4.73
  [1986cc42] Unitful v1.11.0
  [e88e6eb3] Zygote v0.6.37

] instantiate and ] precompile executed successfully before all of this.

This was JuliaPlots/Makie.jl/issues/1817 .

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the Julia 1.6.1 example with Distributed.addprocs, @distributed, and GLMakie, comparing package loading before and after connecting to the workers. Read the Distributed package-loading path and the referenced JuliaPlots/Makie issue 1817; no repository files or tests are named. Done requires an agreed behavior for packages available only on the local machine and a verified resolution or documented limitation.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.