EpistasisLab / EpistasisLab/tpot
Evaluated pipelines not containing all models
- Dominant language
- Jupyter Notebook
- Stars
- 10.1k
- Forks
- 1.6k
- PR merge metrics
- No merged PRs in 30d
Description
##Context of the issue
So I am trying to "stalk" the score of pipelines during training with warm-start and small runs.
I have managed to do so for both "Classifier" and "Regressor" template, but pipelines start to not appear in evaluated pipelines after I add "Transformer-" to the config
## Process to reproduce the issue
Run the following codes and compare testC and testC2's output.
independent=np.random.randint(100,size=1000)
dependent=np.random.randint(2,size=1000)
X_train, X_test, Y_train, Y_test = train_test_split(
independent, dependent, train_size=0.7, test_size=0.3
)
X_train=X_train.reshape(-1,1)
medium_config = {
"sklearn.linear_model.LogisticRegression": {
"penalty": ["l1", "l2"],
"C": [1e-4, 1e-3, 1e-2, 1e-1, 0.5, 1.0, 5.0, 10.0, 15.0, 20.0, 25.0],
"dual": [False],
},
"sklearn.tree.DecisionTreeClassifier": {
"max_depth": range(1, 21),
"min_samples_split": range(2, 21),
"min_samples_leaf": range(1, 21),
},
"sklearn.ensemble.RandomForestClassifier": {
"n_estimators": np.arange(10, 201, 5),
"max_features": np.arange(0.05, 1.01, 0.05),
"min_samples_split": range(2, 21),
"min_samples_leaf": range(1, 21),
"bootstrap": [True, False],
},
"xgboost.XGBClassifier": {
"objective": ['reg:squarederror'],
"n_estimators": np.arange(10, 201, 5),
"max_depth": range(1, 21),
"learning_rate": [1e-3, 1e-2, 1e-1, 0.5, 1.0],
"subsample": np.arange(0.05, 1, 0.05),
"min_child_weight": range(1, 21),
"nthread": [1],
},
# Transformers
"sklearn.preprocessing.Binarizer": {"threshold": np.arange(0.0, 1.01, 0.05)},
"sklearn.preprocessing.MinMaxScaler": {},
"sklearn.preprocessing.RobustScaler": {},
"sklearn.preprocessing.StandardScaler": {},
}
testC = tpotC(
generations=2,
population_size=30,
verbosity=3,
config_dict=medium_config,
n_jobs=2,
scoring="accuracy",
random_state=123,
use_dask=True,
template="Classifier",
warm_start=True,
)
testC2 = tpotC(
generations=2,
population_size=30,
verbosity=3,
config_dict=medium_config,
n_jobs=2,
scoring="accuracy",
random_state=123,
use_dask=True,
template="Transformer-Classifier",
warm_start=True,
)
scatter = []
pipelineNames=[]
#Change testC to testC2 to see the problem
for i in range(20):
testC.fit(X_train,Y_train)
tuples=list(testC.evaluated_individuals_.items())
tuples.sort(key=lambda x: x[1]["internal_cv_score"], reverse=True)
shownModels = []
count=0
for x in tuples:
pipeline = x[0]
name = pipeline[: pipeline.find("(")]
if name in shownModels:
continue
if(i==0):
pipelineNames.append(name)
scatter.append([])
shownModels.append(name)
description = x[1]
score = description["internal_cv_score"]
scatter[count].append(score)
count += 1
fig,ax=plt.subplots(1,1)
for j in range(len(scatter)):
ax.plot(range(0, len(scatter[j])), scatter[j], "-x", label=pipelineNames[j])
plt.show()
print("run", i, "done")
## Expected result
template Transformer-Classifier to behave like template Classifier, where every time I request for evaluated_pipelines, all models are in there. Also Regressors for the same thing.
## Current result
With Transformer in the template, certain models start to not be included in the evaluated_pipelines_, and the models to be left behind are not constant over different runs.
## Possible fix
No idea actually, help!
## Screenshot
Classifier

Transformer-Classifier

Thank you.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.