EpistasisLab / EpistasisLab/tpot

Evaluated pipelines not containing all models

Open
#982 6 comments 0 reactions 0 assignees View on GitHub
bug question
Dominant language
Jupyter Notebook
Stars
10.1k
Forks
1.6k
PR merge metrics
No merged PRs in 30d

Description

##Context of the issue

So I am trying to "stalk" the score of pipelines during training with warm-start and small runs.
I have managed to do so for both "Classifier" and "Regressor" template, but pipelines start to not appear in evaluated pipelines after I add "Transformer-" to the config

## Process to reproduce the issue

Run the following codes and compare testC and testC2's output.

independent=np.random.randint(100,size=1000)
dependent=np.random.randint(2,size=1000)
X_train, X_test, Y_train, Y_test = train_test_split(
independent, dependent, train_size=0.7, test_size=0.3
)
X_train=X_train.reshape(-1,1)

medium_config = {
"sklearn.linear_model.LogisticRegression": {
"penalty": ["l1", "l2"],
"C": [1e-4, 1e-3, 1e-2, 1e-1, 0.5, 1.0, 5.0, 10.0, 15.0, 20.0, 25.0],
"dual": [False],
},
"sklearn.tree.DecisionTreeClassifier": {
"max_depth": range(1, 21),
"min_samples_split": range(2, 21),
"min_samples_leaf": range(1, 21),
},
"sklearn.ensemble.RandomForestClassifier": {
"n_estimators": np.arange(10, 201, 5),
"max_features": np.arange(0.05, 1.01, 0.05),
"min_samples_split": range(2, 21),
"min_samples_leaf": range(1, 21),
"bootstrap": [True, False],
},
"xgboost.XGBClassifier": {
"objective": ['reg:squarederror'],
"n_estimators": np.arange(10, 201, 5),
"max_depth": range(1, 21),
"learning_rate": [1e-3, 1e-2, 1e-1, 0.5, 1.0],
"subsample": np.arange(0.05, 1, 0.05),
"min_child_weight": range(1, 21),
"nthread": [1],
},
# Transformers
"sklearn.preprocessing.Binarizer": {"threshold": np.arange(0.0, 1.01, 0.05)},
"sklearn.preprocessing.MinMaxScaler": {},
"sklearn.preprocessing.RobustScaler": {},
"sklearn.preprocessing.StandardScaler": {},
}

testC = tpotC(
generations=2,
population_size=30,
verbosity=3,
config_dict=medium_config,
n_jobs=2,
scoring="accuracy",
random_state=123,
use_dask=True,
template="Classifier",
warm_start=True,
)

testC2 = tpotC(
generations=2,
population_size=30,
verbosity=3,
config_dict=medium_config,
n_jobs=2,
scoring="accuracy",
random_state=123,
use_dask=True,
template="Transformer-Classifier",
warm_start=True,
)

scatter = []
pipelineNames=[]

#Change testC to testC2 to see the problem

for i in range(20):
testC.fit(X_train,Y_train)

tuples=list(testC.evaluated_individuals_.items())
tuples.sort(key=lambda x: x[1]["internal_cv_score"], reverse=True)
shownModels = []
count=0

for x in tuples:
pipeline = x[0]
name = pipeline[: pipeline.find("(")]
if name in shownModels:
continue
if(i==0):
pipelineNames.append(name)
scatter.append([])
shownModels.append(name)
description = x[1]
score = description["internal_cv_score"]
scatter[count].append(score)
count += 1
fig,ax=plt.subplots(1,1)
for j in range(len(scatter)):
ax.plot(range(0, len(scatter[j])), scatter[j], "-x", label=pipelineNames[j])
plt.show()
print("run", i, "done")

## Expected result

template Transformer-Classifier to behave like template Classifier, where every time I request for evaluated_pipelines, all models are in there. Also Regressors for the same thing.

## Current result

With Transformer in the template, certain models start to not be included in the evaluated_pipelines_, and the models to be left behind are not constant over different runs.

## Possible fix
No idea actually, help!

## Screenshot
Classifier
![image](https://user-images.githubusercontent.com/42673559/71504297-df5ea400-28b3-11ea-834f-fbb836f0fe48.png)

Transformer-Classifier
![image](https://user-images.githubusercontent.com/42673559/71504306-e685b200-28b3-11ea-85df-541a4edb8885.png)

Thank you.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.