automl / automl/HyperSHAP

Negative shapely values in tunability()?

Open
#39 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
21
Forks
5
PR merge metrics
No merged PRs in 30d

Description

What do negative shapely values mean? Is this a bug?
The way I understood it, they should not be possible.

## Minimal reproduction

```python
import numpy as np
from ConfigSpace import ConfigurationSpace, Integer, Configuration
from hypershap import HyperSHAP, ExplanationTask

rng = np.random.RandomState(4)
cs = ConfigurationSpace(seed=4)
for i in range(15):
cs.add([Integer(f"x{i}", (0, 29), default=15)])
data = []
for _ in range(1000):
vals = {f"x{i}": rng.randint(0, 30) for i in range(15)}
c = Configuration(cs, vals)
cost = sum(vals.values()) + rng.normal(0, 5)
data.append((c, cost))

hshap = HyperSHAP(ExplanationTask.from_data(cs, data), approximation_budget=2**9)
iv = hshap.tunability(order=1, baseline_config=cs.get_default_configuration(), index="FSII")

neg = [(k, v) for k, v in iv.dict_values.items() if k and v < 0]
print(f"Negative values: {neg}")
```
Output:

```
Negative values: [((1,), -1.001706630271433)]
```

Reproduced with:
- `hypershap==0.0.6`
- `shapiq==1.4.1`
- `ConfigSpace==1.2.2`
- `scikit-learn==1.8.0rc1`
- `numpy==2.2.6`

## Possible issues

1. hypershap/utils.py:115 declares `self.colation_cache = {}` but this isn't actually used anywhere. The comment above says: `# cache coalition values to ensure monotonicity for min/max`. Maybe that means monotonicity is not ensured?
2. This issue seemingly appears with many hyperparameters, so maybe it's an issue of the approximator that kicks in with more than 14 HP. Can the approximation produce negative values? If yes maybe clip the approximated negative SI values to 0.

Contributor guide

Open the contributing guide

Research direction

Start with the minimal reproduction for HyperSHAP.tunability(order=1, index="FSII") and inspect hypershap/utils.py around line 115, including the unused colation_cache. Trace how coalition values and approximated interaction values are computed for the 15-parameter case. Done means identifying whether the negative value is expected or a bug and adding or updating coverage for the reported behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.