It seems like the 'success' value in the return of the 'attack' function is overconfident.
- Dominant language
- Python
- Stars
- 3k
- Forks
- 442
- PR merge metrics
- No merged PRs in 30d
Description
It seems like the 'success' value in the return of the 'attack' function is overconfident.
```python
if __name__ == '__main__':
args = parser.parse_args()
model = ModelController(args).get_model().eval()
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
ConvCTrainer.load_model(args, model, device)
data_controller = DataCController(args)
eval_loader = data_controller.get_test_dataloader()
mean, std = data_controller.aug_controller.get_normalize_param()
attack = LinfFastGradientAttack()
fmodel = PyTorchModel(model, bounds = (0, 1), preprocessing = dict(mean = mean, std = std))
epsilons = [0.01, 0.03, 0.1, 0.3, 0.5]
cnt, total = torch.zeros(len(epsilons)).to(device),\
torch.zeros(len(epsilons)).to(device)
correct = torch.zeros(len(epsilons)).to(device)
for _, (images, labels) in enumerate(eval_loader):
images = images.to(device)
labels = labels.to(device)
images = images * std[:, None, None] + mean[:, None, None]
_, advs_list, success = attack(fmodel, images, labels, epsilons = epsilons)
cnt += success.sum(axis = 1)
total += images.shape[0]
for i, advs in enumerate(advs_list):
preds = model(advs).argmax(dim=1)
correct[i] += (preds == labels).sum().item() # Compute accuracy for each epsilon
print(f"Success rate vector: {cnt / total}")
print(f"Accuracy vector for each epsilon: {correct / total}")
```
And the output of this code is
```shell
Success rate vector: tensor([0.3397, 0.4297, 0.5650, 0.6253, 0.6954], device='cuda:0')
Accuracy vector for each epsilon: tensor([0.9872, 0.9851, 0.9676, 0.7457, 0.3955], device='cuda:0')
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at the LinfFastGradientAttack call and its success return, then compare that value with the model predictions computed for each advs in the example. Reproduce the reported success-rate and accuracy vectors with the shown PyTorch evaluation code; done means the returned success values agree with the attack outcome represented by those predictions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning, security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100