automl / automl/ParameterImportance

--marginalize_over_instances option and fanova results

Open
#88 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
78
Forks
19
PR merge metrics
No merged PRs in 30d

Description

Hello,

When I apply fanova on my data with and without specifying "--marginalize_over_instances", the list of the most important parameters are very different. I'm just wondering why it is the case. To my understanding, marginalization over instances for fanova analysis is always done in both cases. If that is not correct and they are meant to be different, which results should I use?

Please find in [attachment](https://github.com/automl/ParameterImportance/files/2585202/data.zip) the example data I'm using. There are 4 parameters and 4985 data points. The data may look a bit artificial because it was tuning data given by irace and was converted to smac output format. The commands I use and the resulting list of important parameters are listed below:

###### pimp -S scenario.txt -H runhistory.json -M fanova
LL_static_crossOverBias 84.193
LL_static_lda 2.270
LL_static_lambda1 0.750
LL_static_lambda2 0.555

###### pimp -S scenario.txt -H runhistory.json -M fanova --marginalize_over_instances
LL_static_lambda1 69.746
LL_static_lambda2 5.647
LL_static_crossOverBias 3.680
LL_static_lda 0.699

Many thanks,
Nguyen

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce both pimp commands using scenario.txt, runhistory.json, and the attached data.zip, then inspect how fanova handles --marginalize_over_instances. Determine whether the differing rankings are expected or erroneous, and document or correct the behavior so the issue explains the option's semantics and which results users should trust.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.