0x0pointer / 0x0pointer/agent-smith
semgrep and trufflehog silently return empty results instead of failing
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 129
- Forks
- 10
- Avg merge
- 8h 52m
- Merged PRs (30d)
- 5
Description
What happens
scan(tool="semgrep", target=<path>) and scan(tool="trufflehog", target=<path>) both returned:
{"findings": [], "raw": ""}
against a large, real Ruby/JS codebase (a Rails 8 monorepo with committed secrets that were later found by hand: a Rollbar token, two PagerDuty integration keys, and production + staging TimePilot credentials).
Proof that it is a tool failure, not a clean result
Running the same tools against a path that does not exist returned byte-identical output:
scan(tool="semgrep", target="/Users/.../this-path-does-not-exist-zzq123") -> {"findings": [], "raw": ""}
scan(tool="trufflehog", target="/Users/.../this-path-does-not-exist-zzq123") -> {"findings": [], "raw": ""}
A nonexistent target and a secret-laden repo are indistinguishable in the response.
Why this matters
This is the worst failure mode a security scanner can have: a broken scan is reported as a clean scan. I initially reported "static analysis clean" on the basis of these results. Every white-box finding in the engagement ended up coming from manual grep/source reading instead. Any user who trusts the empty result gets a false negative on their entire secret-scanning and SAST coverage.
Expected
- Non-zero exit / missing binary / unreadable target → surface an explicit error in the envelope, and do not return a
findings: []success shape. raw: ""with a valid target should be treated as a failure signal, not success.session(complete)should refuse to count a scanner as "run" when it produced no output at all.
Suggested fix
Propagate the container exit code and stderr into the envelope; add a smoke test that runs each scanner against a fixture containing a known planted secret and asserts len(findings) > 0.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the scan(tool="semgrep"/"trufflehog") entry point and the session(complete) handling described in the issue. Reproduce the identical empty response for a nonexistent target, then run each scanner against a fixture with a planted secret. Done means exit codes and stderr surface as errors, empty raw output is not reported as success, and the smoke tests detect the secret.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- security, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100