Publish reproducible precision and recall measurements for secret detection
- Dominant language
- Rust
- Stars
- 26
- Forks
- 6
- Avg merge
- 1h 13m
- Merged PRs (30d)
- 384
Description
## Source
Intermediate-presentation feedback repeatedly questioned the reported recall, which benchmark produced it, which secret classes are missed, and how false positives are handled.
## Problem
A security filter cannot be evaluated from a single percentage without a fixed corpus, per-detector results, false-negative examples, false-positive examples, and reproducible commands. Users currently cannot tell what Pentect reliably protects or where manual caution is still required.
## Acceptance criteria
- Pin a public or redistributable benchmark corpus and its exact version.
- Report precision, recall, and F1 separately; do not collapse them into one accuracy number.
- Break results down by secret category and detector/plugin configuration.
- Include low-entropy credentials and natural-language password cases.
- Record representative false negatives and false positives without real credentials.
- Publish the exact command, Pentect version, plugin versions, platform, and configuration.
- Run a bounded regression subset in CI and prevent silent security regressions.
- Clearly distinguish upstream CredSweeper results from Pentect's complete pipeline results.
Contributor guide
Research direction
Start by locating Pentect’s complete secret-detection pipeline, CredSweeper integration, plugin configuration, and existing CI entry points. Define a pinned redistributable corpus and reproducible measurement command, then document per-category precision, recall, F1, false positives, and false negatives. Done means the results distinguish upstream CredSweeper from Pentect, publish the required environment details, and run a bounded regression subset in CI.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- security, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100