automl / automl/ParameterImportance

Fanova, Pairs of Variable Plots, AttributeError: 'CategoricalHyperparameter' object has no attribute 'lower'

Open
#25 26 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
78
Forks
19
PR merge metrics
No merged PRs in 30d

Description

Hi,
If I want to create the fanova plots, the single variable plots work, but pairs fail:

```
>>> python3.5 /home/schirrmr/programs/ParameterImportance/scripts/evaluate.py --scenario_file scenario.txt --history 'runhistories-shared/runhistory*.json' --modus fanova
INFO:Importance:Reading Scenario file and files specified in the scenario
INFO:smac.scenario.scenario.Scenario:Reading scenario file: deep4_scenario.txt
INFO:smac.scenario.scenario.Scenario:Output to PIMP_fanova_2017_06_06_14:20:34
INFO:Importance:Reading Runhistory
INFO:Importance:Combined number of Runhistory data points: 1038
INFO:Importance:Converting Data and constructing Model
INFO:Importance:Setting up Evaluation Method
INFO:Importance:Running evaluation method fANOVA
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/allow_crash.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/data_folder.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/do_batch_norm.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/double_time_convs.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/drop_prob.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/exponential_demean_factor.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/exponential_standardize_factor.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/filter_length_2.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/filter_length_3.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/filter_length_4.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/filter_time_length.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/final_dense_length.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/first_nonlin.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/first_pool_mode.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/last_seizure_train_ms.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/later_nonlin.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/later_pool_mode.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/n_chans.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/n_epochs.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/non_seiz_to_seiz_factor.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/num_filters_2.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/num_filters_3.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/num_filters_4.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/num_filters_spat.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/num_filters_time.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/only_return_exp.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/pool_time_length.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/pool_time_stride.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/resample_mean_pred_ms.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/run_file.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/sample_by_preds.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/sampling_rate.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/save_folder.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/split_first_layer.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/threshold.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/['drop_prob', 'filter_length_2'].png
Traceback (most recent call last):
File "/home/schirrmr/programs/ParameterImportance/scripts/evaluate.py", line 52, in
importance.plot_results(name=os.path.join(save_folder, args.modus))
File "/home/schirrmr/programs/ParameterImportance/pimp/importance/importance.py", line 254, in plot_results
self.evaluator.plot_result(name)
File "/home/schirrmr/programs/ParameterImportance/pimp/evaluator/fanova.py", line 68, in plot_result
vis.create_all_plots(name)
File "/home/schirrmr/.local/lib/python3.5/site-packages/fanova/visualizer.py", line 57, in create_all_plots
self.plot_pairwise_marginal(combi, **kwargs)
File "/home/schirrmr/.local/lib/python3.5/site-packages/fanova/visualizer.py", line 114, in plot_pairwise_marginal
lower_bound = self.cs_params[p].lower
AttributeError: 'CategoricalHyperparameter' object has no attribute 'lower'
```

If you need more context: http://nbviewer.jupyter.org/gist/robintibor/6dad6629987cd66b060503bc43e4e2bc/Hyperparameter_June_Report.ipynb

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the failure with scripts/evaluate.py and the provided scenario and runhistory files, then trace plotting through pimp/importance/importance.py and pimp/evaluator/fanova.py into fanova/visualizer.py. Confirm that single-variable and pairwise fANOVA plots are all generated without the reported CategoricalHyperparameter AttributeError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.