EpistasisLab / EpistasisLab/tpot

AutoMLs Benchmark. Why TPOT so bad?

Open
#1,100 1 comment 0 reactions 0 assignees View on GitHub
question
Dominant language
Jupyter Notebook
Stars
10.1k
Forks
1.6k
PR merge metrics
No merged PRs in 30d

Description

I made Benchmark AutoML libs, and TPOT showed very poor results, even worse than the usual CatBoost with standard parameters!
https://github.com/Alex-Lekov/AutoML-Benchmark/
I run the benchmark in docker - so you can easily reproduce it

here is the code from the benchmark:

```
automl = TPOTClassifier(max_time_mins=(TIME_LIMIT//60),
scoring='roc_auc',
verbosity=1,
random_state=RANDOM_SEED, )

automl.fit(X_train, y_train,)

try:
predictions = automl.predict_proba(X_test)
except RuntimeError:
predictions = automl.predict(X_test)


# TPOT make a different predictions format, depending on the algorithm :(

try:
y_test_predict_proba = predictions[:,1]
except IndexError:
y_test_predict_proba = predictions

y_test_predict = automl.predict(X_test)

print('AUC: ', roc_auc_score(y_test, y_test_predict_proba))
```

Is the code correct? (I do not adjust the advanced parameters, since AutoML, in theory, should pick everything up by itself, that's why it is AutoML)

> if we specify scoring = 'roc_auc', does it start optimizing for AUC for sure?

Please, tell me what am I doing wrong? Maybe I am using the library "incorrectly"? or is this a real result and the library is really that bad?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the linked AutoML-Benchmark repository and reproduce the TPOTClassifier run in Docker using the code shown here. Check whether scoring='roc_auc' and the predict_proba/predict handling match the benchmark's intended evaluation, then document whether the result indicates misuse or a TPOT problem.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python, scikit-learn
Domain
machine-learning, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.