tensorflow / tensorflow/privacy

Intuition behind evaluating attack performance on data not in the training set

Open
#140 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2k
Forks
477
Avg merge
22h 12m
Merged PRs (30d)
1

Description

It seems like when creating the attack data, those in the training set are set as 0 and not in the training set are set as 1:
https://github.com/tensorflow/privacy/blob/caf71c11bcdf3f6301db6916a37901b22798c992/tensorflow_privacy/privacy/membership_inference_attack/models.py#L78-L79
https://github.com/tensorflow/privacy/blob/caf71c11bcdf3f6301db6916a37901b22798c992/tensorflow_privacy/privacy/membership_inference_attack/membership_inference_attack.py#L85-L88
In this way, the attack performance (auc and advantage) is evaluated on the data that are not in the training set instead of data in the training set, as pos_label for tpr and fpr in metrics.roc_curve is 1 by default.

I feel like most lieterature evaluates attack performance on the data in the training set instead as people want to see how well an adversary can distinguish if a sample is in the training set. Is there a specific reason for evaluating on the data that are not in the training set here?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the linked sections of tensorflow_privacy/privacy/membership_inference_attack/models.py and membership_inference_attack.py, then trace how labels reach metrics.roc_curve and its default pos_label. Done means establishing whether the label convention intentionally evaluates non-members or whether the attack metrics should represent membership in the training set.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.