alteryx / alteryx/evalml

time_budget parameter might not be strict enough for harder problems

Aperta
#3,827 1 commento 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Python
Stelle
850
Fork
96
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

Hello,

I have been using EvalML for a while and now I tried running analyses on more complex data, while setting a small `time_budget` value to observe how it behaves.

evaml version: `0.59.0`

Analysis settings: Default, with variant `time_budget` (given below)

1. [gisette](https://www.openml.org/search?type=data&status=active&id=41026)
Dimensions: (7000, 5001)
**Time elapsed: 23:51**
Models Trained: naive LR
`time_budget`: 60 (seconds)

2. [QSAR-TID-11109](https://www.openml.org/search?type=data&status=active&id=3915)
Dimensions: (1976, 1025)
Time elapsed: 01:22
Models Trained: naive LR RF
`time_budget`: 60 (seconds)

3. [covertype](https://www.openml.org/search?type=data&status=active&id=1596)
Dimensions: (581012, 55)
Time elapsed: 01:19
Models Trained: naive LR
`time_budget`: 60 (seconds)

4. [AP_Colon_Kidney](https://www.openml.org/search?type=data&status=active&id=1137)
Dimensions: (546, 10936)
**Time elapsed: 35:45**
Models Trained: naive LR
`time_budget`: 120 (seconds)

It is evident in cases **1** and **4** that Evalml created the naive model and then procceeded with the creation of the linear one. The problem is the `Time elapsed` is much bigger than the `time_budget`.
My questions are:
* Is this supposed to happen?
* If yes, is there a way to make the `time_budget` stricter and actually stop at or close to the given `time_budget`?
* What is the reasoning behind this? Perhaps that at least one non-naive model must be trained to complete the analysis?

From my understanding, once a model starts training (within the `time_budget`), it cannot stop. Still, this could pose an issue in tight `time_budget` scenarios. Moreover, the user might be baffled as to why the analysis is not ending.

Regards,
Iordanis Xanthopoulos

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.