JuliaAI / JuliaAI/MLJBase.jl

Something not right with serialisation of python (sk-learn) models

Open
#435 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Julia
Stars
163
Forks
46
Avg merge
1d 18h
Merged PRs (30d)
5

Description

using MLJ

model = @load RandomForestClassifier pkg=ScikitLearn

data = load_iris()
y, X = unpack(data, ==(:target), n->true, rng=1234)

mach = machine(model, X, y) |> fit!
MLJ.save("junk.jlso", mach)

## NEW REPL SESSION

using MLJ

model = @load RandomForestClassifier pkg=ScikitLearn

mach_predict_only = machine("junk.jlso") 

Sometimes the last expression hangs and if I interrupt Julia crashes. Sometimes it works fine. Same is true for JLSO.load("junk.jlso"). Anyone able to recreate?

(junk) pkg> st
Status `~/Dropbox/Julia7/MLJ/MLJ/sandbox/junk/Project.toml`
  [9da8a3cd] JLSO v2.3.2
  [add582a8] MLJ v0.14.1
  [5ae90465] MLJScikitLearnInterface v0.1.6

julia> versioninfo()
Julia Version 1.5.1
Commit 697e782ab8 (2020-08-25 20:08 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin19.5.0)
  CPU: Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-9.0.1 (ORCJIT, skylake)
Environment:
  JULIA_LTS_PATH = /Applications/Julia-1.0.app/Contents/Resources/julia/bin/julia
  JULIA_PATH = /Applications/Julia-1.5.app/Contents/Resources/julia/bin/julia
  JULIA_NUM_THREADS = 5

It may be that the python objects bound to fitresult are not reliably persistent and we need to do implement custom MMI.save and MMI.restore along the lines of XGBoost.

Assuming ScikitLearn.jl exposes the python serialization, this ought to be easy to do.

@OkonSamuel Thoughts?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the provided MLJ.save and machine("junk.jlso") sequence in a fresh Julia REPL using the listed Julia, MLJ, JLSO, and ScikitLearnInterface versions. Inspect the ScikitLearn interface's serialization behavior and compare it with the MMI.save and MMI.restore implementation linked for XGBoost. Done means loading the saved model completes reliably without hanging or crashing.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia, python, scikit-learn
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.