JuliaAI / JuliaAI/MLJBase.jl

taking performance issue more seriously

Open
#309 16 comments 5 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

brainstorm performance
Dominant language
Julia
Stars
163
Forks
46
Avg merge
1d 18h
Merged PRs (30d)
5

Description

I know MLJ will definitely have some overhead since it's wraps other code. But i believe this overhead can be reduced below the current level with careful design considerations. Avoidable overhead which are neglected may come to hunt us when the code is scaled. There are a few things i discovered which are important.

  1. As pointed out in issue #151 selectrows. This causes overhead when used in evaluate method which calls this method a lot of times depending on the repeats parameter.
  2. In the MLJBase.fit method, the matrix method (which to my understanding copies data) is called on a given table X of course this isn't bad if i call this method only once using the same data changing a couple of model parameters. This becomes important in the evaluate method which calls fit a couple of times depending on repeats .(If X is larger this isn't nice). (I don't think update method does enough justice). Also other methods call these methods repeatedly copying X in each case
  3. The return type to MLJBase.predict for probabilistic Classifiers.( I can't find the link to the issue)

These are just some of the point. I believe there are other things which affects scalability. It is better we start treating this issues more serious. Imagine a case where One wants to embark on a kaggle competition on a large dataset only to find out that the overhead is just unbearable in this case). You may correct me perhaps i'm missing something.
@ablaom , @tlienart

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by profiling evaluate and its repeated calls to selectrows and MLJBase.fit, then inspect the matrix call on X and the return type of MLJBase.predict for probabilistic classifiers. Review issue #151 and the existing comment discussion to identify an agreed scope. Done should mean a defined set of bottlenecks is addressed and benchmark results show reduced overhead without changing expected behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
machine-learning, performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.