NVIDIA / NVIDIA/SkillSpector

Creates false positives on pure text-based content

Open
#37 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
17.9k
Forks
1.5k
Avg merge
5d 10h
Merged PRs (30d)
66

Description

I ran Opus 4.8 on all the skills in my Github repo (https://github.com/francoisdp/skills) it came up with the following report:

All scans are complete. Here are the results before I change any file or push anything. None of the eight skills ships executable code, so every finding is a
pattern match against prose, a LICENSE file, or template comments.

SkillSpector results (the 8 deployed skills)

Skill Score Tool verdict Findings
deep-research 5 SAFE 1 LOW (LICENSE boilerplate)
study-note 5 SAFE 1 LOW (LICENSE)
youtube2article 5 SAFE 1 LOW (LICENSE)
branded-document-toolkit 15 SAFE 1 LOW (LICENSE), 1 MEDIUM (README)
model-router 35 CAUTION 1 HIGH, 1 MEDIUM
anti-ai-writing 65 DO_NOT_INSTALL 2 HIGH, 1 MEDIUM, 1 LOW
deploy-skill 65 DO_NOT_INSTALL 1 CRITICAL, 1 MEDIUM, 1 LOW
marp-deck 80 DO_NOT_INSTALL 3 HIGH, 1 LOW

My assessment of the flagged items

I read every HIGH and CRITICAL line in the source. They are all false positives produced by the scanner pattern-matching documentation and template text:

  • deploy-skill — CRITICAL "Harmful Content Injection" at README.md:33. The line is ordinary prose describing the /deploy-skill trigger and that the skill
    refuses to publish another person's work. Nothing harmful.
  • marp-deck — 3× HIGH "Hidden Instructions" in templates. These are the Marp HTML comment blocks (<!-- Template: default … Replace [TITLE] -->). Legitimate
    template authoring comments, not hidden directives.
  • anti-ai-writing — HIGH "Direct Prompt Extraction" at SKILL.md:916 is a section header about output types. HIGH "Tool Parameter Abuse" was matched inside the
    binary bytes of a PDF example.
  • model-router — HIGH "Memory Manipulation" at SKILL.md:81 is the instruction describing what context to pass to a worker subagent, including reading CLAUDE.md.
    Normal dispatch description.
  • The LOW at LICENSE:16 appears on all eight. It is the MIT licence text.

In substance, all eight skills are documentation-only, with no executable scripts, no network calls, no credential access, and no persistence mechanism. The tool
over-flags instruction-heavy skills. SkillSpector has no suppression or baseline flag, so I cannot make the raw verdict turn green for the four flagged skills
without editing the skills themselves.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the report against the linked francoisdp/skills repository and inspect the cited matches, including README.md:33, SKILL.md:81 and :916, LICENSE:16, and the Marp HTML comments. Compare scanner behavior on prose, templates, license text, and binary example content; the issue is resolved when these documented false positives are handled without hiding genuine findings.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
security
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.