Add `TestOnly` resampling strategy?
Nobody has claimed this yet.
- Dominant language
- Julia
- Stars
- 163
- Forks
- 46
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 5
Description
Sometimes you have trained a supervised machine on some data and you want to evaluate on some holdout set without retraining. Using evaluate!(..., resamping=Holdout()) doesn't allow this, so you would need to manually predict, and apply each metric to the prediction and test target, which is inconvenient, especially if you are tracking multiple metrics.
The idea of a TestOnly reampling strategy is that evaluate!(mach, resampling=TestOnly(), rows=test, measures=...) automates this: we assume mach is already trained (or throw an exception) and just evaluate the specified measures on predictions on the test rows.
Implementation looks pretty simple: train_test_pairs(::TestOnly, rows) = [(Int[], rows),] (i.e. empty train) and in evaluate! an empty train set will suppress training.
(It would be very convenient and natural that specifying no resampling strategy would fall back to TestOnly(), as in:
fit!(mach, rows=train)
evaluate!(mach, rows=test, measure=l2)
but that would be technically breaking - the current fallback is CV(). )
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the resampling entry point train_test_pairs and the evaluate! implementation, using the proposed TestOnly strategy and empty training rows as the scope. Check how trained machines, prediction rows, and measures are handled, then add coverage showing evaluation on rows=test without retraining and an exception for an untrained machine.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100