time_budget parameter might not be strict enough for harder problems
- Lenguaje dominante
- Python
- Estrellas
- 850
- Forks
- 96
- Métricas de merge de PR
- Sin PR fusionados en 30 d
Descripción
Hello,
I have been using EvalML for a while and now I tried running analyses on more complex data, while setting a small `time_budget` value to observe how it behaves.
evaml version: `0.59.0`
Analysis settings: Default, with variant `time_budget` (given below)
1. [gisette](https://www.openml.org/search?type=data&status=active&id=41026)
Dimensions: (7000, 5001)
**Time elapsed: 23:51**
Models Trained: naive LR
`time_budget`: 60 (seconds)
2. [QSAR-TID-11109](https://www.openml.org/search?type=data&status=active&id=3915)
Dimensions: (1976, 1025)
Time elapsed: 01:22
Models Trained: naive LR RF
`time_budget`: 60 (seconds)
3. [covertype](https://www.openml.org/search?type=data&status=active&id=1596)
Dimensions: (581012, 55)
Time elapsed: 01:19
Models Trained: naive LR
`time_budget`: 60 (seconds)
4. [AP_Colon_Kidney](https://www.openml.org/search?type=data&status=active&id=1137)
Dimensions: (546, 10936)
**Time elapsed: 35:45**
Models Trained: naive LR
`time_budget`: 120 (seconds)
It is evident in cases **1** and **4** that Evalml created the naive model and then procceeded with the creation of the linear one. The problem is the `Time elapsed` is much bigger than the `time_budget`.
My questions are:
* Is this supposed to happen?
* If yes, is there a way to make the `time_budget` stricter and actually stop at or close to the given `time_budget`?
* What is the reasoning behind this? Perhaps that at least one non-naive model must be trained to complete the analysis?
From my understanding, once a model starts training (within the `time_budget`), it cannot stop. Still, this could pose an issue in tight `time_budget` scenarios. Moreover, the user might be baffled as to why the analysis is not ending.
Regards,
Iordanis Xanthopoulos
Guía de contribución
Evaluación
Este issue todavía no se ha evaluado.