albermax / albermax/innvestigate

Smoothgrad outputs only zeros for a batch of inputs

Open
#246 0 comments 0 reactions 1 assignee Claimed by @adrhill View on GitHub
bug
Dominant language
Python
Stars
1.3k
Forks
230
PR merge metrics
No merged PRs in 30d

Description

Hello!

I tried to analyze a batch of inputs with the 'smoothgrad' method and `neuron_selection_mode = "index"` with your package iNNvestigate (version 1.0.9). Surprisingly, only the output for the first input in the batch made any sense, because the gradients for all other values were set to 0. I think there's something wrong with your code.

Here is a small example:
I generated a classification dataset with `sklearn` with 5 input features and 2 classes. On this dataset, I trained a keras model with 3 dense layers. Then I removed the `softmax` activation and created the analyzers for the methods 'gradient' and 'smoothgrad'. I also set the noise scale (`noise_scale = 1e-8`) to a very small value for the `smoothgrad` method, which makes the results almost identical to the normally calculated gradients. Below you can find the code for my example:

```python
import numpy as np
np.random.seed(1234)

from sklearn.datasets import make_classification

import keras # version 2.2.4
import innvestigate # version 1.0.9
import innvestigate.utils as iutils

n_inputs = 5
n_classes = 2
n_samples = 512

# Generate classification data
data_x, data_y = make_classification(n_samples=n_samples, n_features=n_inputs, n_classes=n_classes)
data_y = keras.utils.to_categorical(data_y)

# Define keras model and fit it
model = keras.models.Sequential([
keras.layers.Dense(16, activation="relu", input_shape=(n_inputs,)),
keras.layers.Dense(8, activation="relu"),
keras.layers.Dense(n_classes, activation="softmax"),
])
model.compile(loss="categorical_crossentropy", optimizer="adam", metrics=["accuracy"])
model.fit(data_x, data_y, epochs=25, batch_size=128)

# Start the analysis with iNNvestigate
model = iutils.keras.graph.model_wo_softmax(model)

# Set the noise level sufficiently low such that almost identical values should come out for the methods
# 'gradient' and 'smoothgrad'
noise_scale = 1e-8

# Define the analyzer for each method
analyzer_smoothgrad = innvestigate.create_analyzer("smoothgrad",
model,
noise_scale = noise_scale,
neuron_selection_mode="index")
analyzer_gradient = innvestigate.create_analyzer("gradient",
model,
neuron_selection_mode="index")

# One input for output neuron '0'
result_smoothgrad = analyzer_smoothgrad.analyze(data_x[1:2,:], 0)
result_gradient = analyzer_gradient.analyze(data_x[1:2,:], 0)
print("Mean squared error (one input): {:4f}".format(np.mean((result_gradient - result_smoothgrad)**2)))

# Multiple inputs for output neuron '0'
result_smoothgrad = analyzer_smoothgrad.analyze(data_x[1:3,:], 0)
result_gradient = analyzer_gradient.analyze(data_x[1:3,:], 0)
print("Mean squared error (multiple inputs): {:4f}".format(np.mean((result_gradient - result_smoothgrad)**2)))

print("Result smoothgrad:")
print(result_smoothgrad)

print("Result gradient:")
print(result_gradient)
```
Here is the output from the example and as you can see the output of 'smoothgrad' for the second input makes no sense:
```bash
Mean squared error (one input): 0.000000
Mean squared error (multiple inputs): 0.225435
Result smoothgrad:
[[ 0.50317013 0.72440994 0.59556705 -1.5030893 -0.39739025]
[ 0. 0. 0. 0. 0. ]]
Result gradient:
[[ 0.5031703 0.72440976 0.5955669 -1.5030882 -0.39739007]
[ 0.46457082 -0.31459585 0.5895394 -1.172667 -0.46567515]]
```
I hope I didn't make a mistake and that I could help you to improve this wonderful package.

Best regards,
Niklas

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.