microsoft / microsoft/FLAML

How to handle metric for TransformersEstimator

Open
#773 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
4.4k
Forks
565
Avg merge
5d 8m
Merged PRs (30d)
17

Description

Currently metric is handled differently for TransformersEstimator in [ml.py](https://github.com/microsoft/FLAML/blob/main/flaml/ml.py#L585) and [another place in ml.py](https://github.com/microsoft/FLAML/blob/main/flaml/ml.py#L651).

The goal is to remove this special requirement. However, metric is required in TransformersEstimator to compute the intermediate result.

Currently, no other estimator needs the metric info. For TransformersEstimator, one solution is to add the metric argument to the __init__ input argument, just like task=="binary". However, that would make some users existing code throw error if the input signature does not contain ** but only task. Should we add this argument to __init__? Or is there other solution that can allow us to evaluate the intermediate results?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.