Add protocol for handling X_val, y_val
Open
- Dominant language
- Jupyter Notebook
- Stars
- 4.4k
- Forks
- 565
- Avg merge
- 5d 8m
- Merged PRs (30d)
- 17
Description
The handling for X_val, y_val for NLP model needs to be taken care of differently. In NLP mode (and only in NLP), one trial can have multiple checkpoints. If a trial has more than 1 checkpoints, how to evaluate the trial's perf for cross-validation is not straightforward.
The following protocol should be added:
If NLP is not under the cross-validation model: requires X_val, y_val for TransformersEstimator.fit, raise an exception if the input does not contain X_val, y_val.
If cross-validation: only evaluate at the end of the training.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.