NVIDIA / NVIDIA/SkillSpector

Coverage is computed but never affects the exit code: a partially-blind scan is indistinguishable from a complete one in CI

Open
#389 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
17.9k
Forks
1.5k
Avg merge
5d 10h
Merged PRs (30d)
66

Description

Summary

analysis_completeness already carries everything needed to know how much of a skill was
inspected: coverage_percent, is_complete, ledger_exceptions with reason codes. It is a
good design and it is why the numbers below exist at all.

But it stops at the JSON boundary. skillspector scan exits 1 only when
risk_score > RISK_THRESHOLD, and 2 on error. A scan that inspected 86 % of the components
exits 0 exactly like one that inspected 100 %.
In CI, where the exit code is the whole
interface, the two are indistinguishable.

Measured

91 reports, Claude Code skill marketplaces on one workstation:

is_complete: false 88 / 91
worst coverage_percent 86.4 %
reasons binary_content 665, llm_batch_failed 344, size_limit 91, syntax_error 1

The size_limit bucket is the one that changes conclusions: 73 distinct .js files above
MAX_FILE_CHARS were never opened by any static analyzer. They are executable files inside
installed skills. The ledger records this correctly — nothing is hidden — but no automated
gate can act on it.

Padding a payload past 1 000 000 characters is not a sophisticated evasion.

Suggestion

Two opt-in flags, so nothing changes for existing users:

--min-coverage FLOAT     non-zero exit if coverage_percent < FLOAT
--fail-on-incomplete     non-zero exit if is_complete is false

A distinct exit code (say 3) would keep "risk found" and "could not look" separable — they
lead to different actions.

One caveat from having built this in a wrapper: an absolute threshold fires on 88 reports out
of 91 on day one, and then gets switched off. Accepting the current coverage into a baseline
and failing on regression proved far more durable. That may be more than SkillSpector wants
to own; a plain --min-coverage already unblocks the CI case.

Happy to send a PR.

Environment

SkillSpector 2.5.1 and main @ 2.9.5, Linux, Python 3.13.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the skillspector scan exit-code path and follow analysis_completeness through the JSON boundary. Review how the risk threshold and error exits are handled, then trace option parsing for the proposed flags. Done means opt-in coverage gates preserve existing behavior and distinguish incomplete scans from risk findings with a separate non-zero exit code.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ci-cd, cli, security
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.