How to handle metric for TransformersEstimator
- Dominant language
- Jupyter Notebook
- Stars
- 4.4k
- Forks
- 565
- Avg merge
- 5d 8m
- Merged PRs (30d)
- 17
Description
Currently metric is handled differently for TransformersEstimator in [ml.py](https://github.com/microsoft/FLAML/blob/main/flaml/ml.py#L585) and [another place in ml.py](https://github.com/microsoft/FLAML/blob/main/flaml/ml.py#L651).
The goal is to remove this special requirement. However, metric is required in TransformersEstimator to compute the intermediate result.
Currently, no other estimator needs the metric info. For TransformersEstimator, one solution is to add the metric argument to the __init__ input argument, just like task=="binary". However, that would make some users existing code throw error if the input signature does not contain ** but only task. Should we add this argument to __init__? Or is there other solution that can allow us to evaluate the intermediate results?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.