dotnet / dotnet/machinelearning-modelbuilder

[AutoML,CLI,ModelBuilder] Exploration Time vs Epochs + Loss/AUC (Enhancement)

Abierto
#699 4 comentarios 3 reacciones 0 asignados Ver en GitHub
Feature request
Lenguaje dominante
Dockerfile
Estrellas
285
Forks
66
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

This is more of a feature enhancement rather than an issue. Please let me know if this is the correct venue to submit this to.

In both the Model Builder GUI and the CLI for AutoML, the only option to customize training time is through the **max exploration time** setting. Example of CLI options:

![image](https://user-images.githubusercontent.com/12853868/57419929-a7d2a880-71b9-11e9-908e-1ce2d41e84e2.png)

This is a really neat way of changing things around instead of using epochs, using a time-based approach. But this could lead to undertraining a data set fairly easily, especially large ones, or produce a model that due to the fact that it hasn't been trained long enough (through several epochs throughout the entire dataset) didn't produce a good accuracy rate as expected.

I'm going to take a wild guess (can't find documentation for this) that the time based approach takes into account AUC or Loss as a criteria (or another common metric) to stop the training, is that correct? In other words if I have a small dataset and I set a training time of 4 hours, but the AUC or Loss didn't improve over X iterations it calls it a day and gives me the best result, is that correct?

It would be nice if the AutoML function (Model Builder, CLI and API) provided a way to (in addition to the time-based approach, or as an alternative to):

1. Set epochs for training rather than time based approach.
2. Set metric to look for when stopping a training, i.e.: auc, loss, etc.
3. Set thresholds of metrics to stop a training. i.e.: if after X number of epochs it doesn't improve, call it a day.

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Línea de trabajo

Comienza revisando los controles de entrenamiento de AutoML expuestos por la GUI, la CLI y la API de Model Builder; después, verifica cómo el tiempo máximo de exploración determina actualmente la detención y si se utiliza AUC o loss. Define el comportamiento esperado para los límites de épocas, las métricas de detención y los umbrales de métricas, y añade cobertura para cada superficie antes de considerar completada la mejora.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Área
api, cli, machine-learning
Tipo de issue
Nueva funcionalidad
Dificultad
5/5
Tiempo estimado
Más de una semana
Estado de actividad
Estancado
Claridad
Bastante claro
Aptitud para principiantes
25/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.