EdamAme-x / EdamAme-x/pentect

Publish reproducible precision and recall measurements for secret detection

Open
#348 0 comments 0 reactions 0 assignees View on GitHub
enhancement priority:P2
Dominant language
Rust
Stars
26
Forks
6
Avg merge
1h 13m
Merged PRs (30d)
384

Description

## Source

Intermediate-presentation feedback repeatedly questioned the reported recall, which benchmark produced it, which secret classes are missed, and how false positives are handled.

## Problem

A security filter cannot be evaluated from a single percentage without a fixed corpus, per-detector results, false-negative examples, false-positive examples, and reproducible commands. Users currently cannot tell what Pentect reliably protects or where manual caution is still required.

## Acceptance criteria

- Pin a public or redistributable benchmark corpus and its exact version.
- Report precision, recall, and F1 separately; do not collapse them into one accuracy number.
- Break results down by secret category and detector/plugin configuration.
- Include low-entropy credentials and natural-language password cases.
- Record representative false negatives and false positives without real credentials.
- Publish the exact command, Pentect version, plugin versions, platform, and configuration.
- Run a bounded regression subset in CI and prevent silent security regressions.
- Clearly distinguish upstream CredSweeper results from Pentect's complete pipeline results.

Contributor guide

Open the contributing guide

Research direction

Start by locating Pentect’s complete secret-detection pipeline, CredSweeper integration, plugin configuration, and existing CI entry points. Define a pinned redistributable corpus and reproducible measurement command, then document per-category precision, recall, F1, false positives, and false negatives. Done means the results distinguish upstream CredSweeper from Pentect, publish the required environment details, and run a bounded regression subset in CI.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
security, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.