google-deepmind / google-deepmind/synthid-text
Proposal: reproducible detector calibration and robustness benchmark
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 104
- PR merge metrics
- No merged PRs in 30d
Description
## Proposal
Add a small, reproducible benchmark harness for calibrating SynthID Text detectors across output lengths and common benign text transformations.
The README already notes that users of the Weighted Mean detector should empirically or theoretically compute thresholds for their desired false-positive rate at specific token lengths. At the moment, each user has to build that calibration workflow independently.
A first-party benchmark could standardize that process by taking a fixed corpus of watermarked and unwatermarked samples and reporting detector quality stratified by effective token length.
A first version could include:
- fixed evaluation splits for watermarked and unwatermarked text;
- length buckets such as 32, 64, 128, 256, and 512 effective detector tokens;
- ROC/AUC plus TPR at selected FPR targets;
- threshold calibration per length bucket;
- confidence intervals from deterministic bootstrap resampling;
- a small set of benign transformations such as prefix/suffix text, whitespace/punctuation normalization, and truncation;
- machine-readable JSON output so results can be compared across detector changes.
Conceptually:
```bash
python -m synthid_text.benchmark_detector --detector weighted_mean --lengths 32,64,128,256,512 --fpr 0.001,0.01
```
could produce both a human-readable table and a versioned JSON artifact containing thresholds and metrics.
## Motivation
Detector scores are not directly comparable across text lengths, and the repository documentation already recommends length-specific calibration. A shared benchmark would make that recommendation reproducible and would give contributors a stable way to answer questions such as:
- Did a detector change improve recall at the same false-positive rate?
- Does calibration drift at short or long outputs?
- Are thresholds stable across a fixed reference corpus?
- Does a change improve one detector while degrading another?
This would also provide a regression signal for the Mean, Weighted Mean, and Bayesian detector implementations without requiring exact per-example scores to remain frozen.
The intent is research reproducibility, not a production certification or adversarial-evasion suite. The initial transformations should stay simple and benign, and the benchmark should report aggregate detection behavior rather than provide optimization guidance for removing watermarks.
## Scope
I would keep the initial contribution deliberately small:
1. define a versioned benchmark-result schema;
2. add reusable calibration/metric utilities;
3. include one small public-safe evaluation split or deterministic fixture generator;
4. benchmark the Weighted Mean detector first;
5. add tests and documentation;
6. leave Bayesian-detector training comparisons as a follow-up if maintainers want them.
I searched current issues for detector calibration, robustness benchmarking, length-stratified thresholds, and benchmark harnesses and did not find an equivalent proposal.
If this direction fits the repository, I would be happy to prototype the harness, metrics, fixtures, tests, and documentation.
Contributor guide
Research direction
Start with the README's guidance on length-specific calibration and inspect the Mean, Weighted Mean, and Bayesian detector implementations. Define the versioned result schema, reusable metric utilities, fixture or evaluation split, and command-line entry point described in the proposal. Done means the Weighted Mean benchmark reports length-stratified metrics, calibrated thresholds, deterministic confidence intervals, JSON output, tests, and documentation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100