automl / automl/ParameterImportance
--marginalize_over_instances option and fanova results
- Dominant language
- Python
- Stars
- 78
- Forks
- 19
- PR merge metrics
- No merged PRs in 30d
Description
Hello,
When I apply fanova on my data with and without specifying "--marginalize_over_instances", the list of the most important parameters are very different. I'm just wondering why it is the case. To my understanding, marginalization over instances for fanova analysis is always done in both cases. If that is not correct and they are meant to be different, which results should I use?
Please find in [attachment](https://github.com/automl/ParameterImportance/files/2585202/data.zip) the example data I'm using. There are 4 parameters and 4985 data points. The data may look a bit artificial because it was tuning data given by irace and was converted to smac output format. The commands I use and the resulting list of important parameters are listed below:
###### pimp -S scenario.txt -H runhistory.json -M fanova
LL_static_crossOverBias 84.193
LL_static_lda 2.270
LL_static_lambda1 0.750
LL_static_lambda2 0.555
###### pimp -S scenario.txt -H runhistory.json -M fanova --marginalize_over_instances
LL_static_lambda1 69.746
LL_static_lambda2 5.647
LL_static_crossOverBias 3.680
LL_static_lda 0.699
Many thanks,
Nguyen
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce both pimp commands using scenario.txt, runhistory.json, and the attached data.zip, then inspect how fanova handles --marginalize_over_instances. Determine whether the differing rankings are expected or erroneous, and document or correct the behavior so the issue explains the option's semantics and which results users should trust.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100