alteryx / alteryx/evalml

time_budget parameter might not be strict enough for harder problems

Abierto
#3,827 1 comentario 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Python
Estrellas
850
Forks
96
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

Hello,

I have been using EvalML for a while and now I tried running analyses on more complex data, while setting a small `time_budget` value to observe how it behaves.

evaml version: `0.59.0`

Analysis settings: Default, with variant `time_budget` (given below)

1. [gisette](https://www.openml.org/search?type=data&status=active&id=41026)
Dimensions: (7000, 5001)
**Time elapsed: 23:51**
Models Trained: naive LR
`time_budget`: 60 (seconds)

2. [QSAR-TID-11109](https://www.openml.org/search?type=data&status=active&id=3915)
Dimensions: (1976, 1025)
Time elapsed: 01:22
Models Trained: naive LR RF
`time_budget`: 60 (seconds)

3. [covertype](https://www.openml.org/search?type=data&status=active&id=1596)
Dimensions: (581012, 55)
Time elapsed: 01:19
Models Trained: naive LR
`time_budget`: 60 (seconds)

4. [AP_Colon_Kidney](https://www.openml.org/search?type=data&status=active&id=1137)
Dimensions: (546, 10936)
**Time elapsed: 35:45**
Models Trained: naive LR
`time_budget`: 120 (seconds)

It is evident in cases **1** and **4** that Evalml created the naive model and then procceeded with the creation of the linear one. The problem is the `Time elapsed` is much bigger than the `time_budget`.
My questions are:
* Is this supposed to happen?
* If yes, is there a way to make the `time_budget` stricter and actually stop at or close to the given `time_budget`?
* What is the reasoning behind this? Perhaps that at least one non-naive model must be trained to complete the analysis?

From my understanding, once a model starts training (within the `time_budget`), it cannot stop. Still, this could pose an issue in tight `time_budget` scenarios. Moreover, the user might be baffled as to why the analysis is not ending.

Regards,
Iordanis Xanthopoulos

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.