Coverage is computed but never affects the exit code: a partially-blind scan is indistinguishable from a complete one in CI
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 1.5k
- Avg merge
- 5d 10h
- Merged PRs (30d)
- 66
Description
Summary
analysis_completeness already carries everything needed to know how much of a skill was
inspected: coverage_percent, is_complete, ledger_exceptions with reason codes. It is a
good design and it is why the numbers below exist at all.
But it stops at the JSON boundary. skillspector scan exits 1 only when
risk_score > RISK_THRESHOLD, and 2 on error. A scan that inspected 86 % of the components
exits 0 exactly like one that inspected 100 %. In CI, where the exit code is the whole
interface, the two are indistinguishable.
Measured
91 reports, Claude Code skill marketplaces on one workstation:
is_complete: false |
88 / 91 |
worst coverage_percent |
86.4 % |
| reasons | binary_content 665, llm_batch_failed 344, size_limit 91, syntax_error 1 |
The size_limit bucket is the one that changes conclusions: 73 distinct .js files above
MAX_FILE_CHARS were never opened by any static analyzer. They are executable files inside
installed skills. The ledger records this correctly — nothing is hidden — but no automated
gate can act on it.
Padding a payload past 1 000 000 characters is not a sophisticated evasion.
Suggestion
Two opt-in flags, so nothing changes for existing users:
--min-coverage FLOAT non-zero exit if coverage_percent < FLOAT
--fail-on-incomplete non-zero exit if is_complete is false
A distinct exit code (say 3) would keep "risk found" and "could not look" separable — they
lead to different actions.
One caveat from having built this in a wrapper: an absolute threshold fires on 88 reports out
of 91 on day one, and then gets switched off. Accepting the current coverage into a baseline
and failing on regression proved far more durable. That may be more than SkillSpector wants
to own; a plain --min-coverage already unblocks the CI case.
Happy to send a PR.
Environment
SkillSpector 2.5.1 and main @ 2.9.5, Linux, Python 3.13.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the skillspector scan exit-code path and follow analysis_completeness through the JSON boundary. Review how the risk threshold and error exits are handled, then trace option parsing for the proposed flags. Done means opt-in coverage gates preserve existing behavior and distinguish incomplete scans from risk findings with a separate non-zero exit code.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ci-cd, cli, security
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100