automl / automl/ParameterImportance

LPI throws error on forbidden/condition combination

Open
#98 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
78
Forks
19
PR merge metrics
No merged PRs in 30d

Description

This happens e.g. if there are parameters `p1` in `[1, 2]` with default value `1`, `p2` in `[True, False]` and there is a forbidden clause that forbids `p1` to be `1` if `p2 == True`, but at the same time there is a conditional that makes `p1` inactive if `p2 == True`. When using ConfigSpace's [impute_inactive-method](https://github.com/automl/ConfigSpace/blob/24d1bcf5deb301d326acf58eb22186bb812d5f84/ConfigSpace/util.py#L43), it creates a `Configuration` with all values set to there default's if they're `None`. If this happens for a legal configuration `p1 = None, p2 = True`, it becomes an illegal one.

Suggested solution: If the actual configuration is not important (if we just need the vector here), fix it by modifying the configuration space object before imputing the values and delete any forbidden clauses. Analogous to [here](https://github.com/automl/CAVE/pull/226).

```
File "/home/shuki/Documents/ParameterImportance/pimp/importance/importance.py", line 500, in evaluate_scenario
dict_[self.evaluator.name.lower()] = self.evaluator.run()
File "/home/shuki/Documents/ParameterImportance/pimp/evaluator/local_parameter_importance.py", line 211, in run
vector=new_array))
File "ConfigSpace/util.py", line 73, in ConfigSpace.util.impute_inactive_values
File "ConfigSpace/configuration_space.py", line 1067, in ConfigSpace.configuration_space.Configuration.__init__
File "ConfigSpace/configuration_space.py", line 1093, in ConfigSpace.configuration_space.Configuration.is_valid_configuration
File "ConfigSpace/c_util.pyx", line 38, in ConfigSpace.c_util.check_configuration
File "ConfigSpace/c_util.pyx", line 138, in ConfigSpace.c_util.check_configuration
File "ConfigSpace/configuration_space.py", line 847, in ConfigSpace.configuration_space.ConfigurationSpace._check_forbidden
File "ConfigSpace/c_util.pyx", line 28, in ConfigSpace.c_util.check_forbidden
File "ConfigSpace/c_util.pyx", line 35, in ConfigSpace.c_util.check_forbidden
ConfigSpace.exceptions.ForbiddenValueError: Given vector violates forbidden clause (Forbidden: weight_decay == 3.2e-09 && Forbidden: use_weight_decay == False)
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in pimp/importance/importance.py at evaluate_scenario and follow the call into pimp/evaluator/local_parameter_importance.py:211 and ConfigSpace.util.impute_inactive_values. Reproduce the forbidden/conditional case from the report and verify that imputing inactive values no longer rejects the legal configuration; use the linked CAVE pull request as the analogous reference.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.