JuliaAI / JuliaAI/MLJBase.jl

Add `TestOnly` resampling strategy?

Open
#976 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Julia
Stars
163
Forks
46
Avg merge
1d 18h
Merged PRs (30d)
5

Description

Sometimes you have trained a supervised machine on some data and you want to evaluate on some holdout set without retraining. Using evaluate!(..., resamping=Holdout()) doesn't allow this, so you would need to manually predict, and apply each metric to the prediction and test target, which is inconvenient, especially if you are tracking multiple metrics.

The idea of a TestOnly reampling strategy is that evaluate!(mach, resampling=TestOnly(), rows=test, measures=...) automates this: we assume mach is already trained (or throw an exception) and just evaluate the specified measures on predictions on the test rows.

Implementation looks pretty simple: train_test_pairs(::TestOnly, rows) = [(Int[], rows),] (i.e. empty train) and in evaluate! an empty train set will suppress training.

(It would be very convenient and natural that specifying no resampling strategy would fall back to TestOnly(), as in:

fit!(mach, rows=train)
evaluate!(mach, rows=test, measure=l2)

but that would be technically breaking - the current fallback is CV(). )

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the resampling entry point train_test_pairs and the evaluate! implementation, using the proposed TestOnly strategy and empty training rows as the scope. Check how trained machines, prediction rows, and measures are handled, then add coverage showing evaluation on rows=test without retraining and an exception for an untrained machine.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.