NVIDIA / NVIDIA/SkillSpector

P6 false positive on HTML Output Rules documentation heading

Open
#512 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
17.9k
Forks
1.5k
Avg merge
5d 10h
Merged PRs (30d)
66

Description

P6 reports a harmless HTML report-formatting heading as a HIGH-severity Direct Prompt Extraction finding.

Reproduced on main at 69dcdfb74487d361ba4c811d088cfdea2ff3a9dc (2.11.2) with this SKILL.md:

---
name: report-example
description: Generate a local HTML report from user-provided data.
---

# Report Example

## HTML Output Rules (Offline-Safe)

- Include figure titles, axis labels, and legends.
- Keep the generated HTML usable without network access.

Run:

skillspector scan ./report-example --no-llm --format json --output report.json

The report contains P6 at SKILL.md:8, matched text Output Rules, severity HIGH, confidence 0.85. The scan completes successfully; this finding contributes a risk score of 21 (CAUTION).

The heading describes the generated report's formatting requirements. It does not request disclosure of the assistant's system prompt or hidden instructions. P6 currently allows both your and system to be absent, so it interprets the noun phrase as an imperative. The existing exception for exactly ## Output Rules (Both Modes) does not cover this heading.

Expected behavior:

  • Recognize unambiguous report-formatting headings without a P6 finding.
  • Preserve actual prompt/rule extraction instructions, including instructions in headings, code blocks, and adjacent text.
  • Preserve finding locations, accounting, and JSON/Markdown/SARIF output correctness.
  • Add paired benign and malicious regressions and verify the complete CLI scan, not only the regex.

Related: #37 covers documentation false positives broadly; #453 addresses a different nominal print rules phrase. This issue tracks the report-formatting heading case specifically.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the P6 scan path using the supplied SKILL.md example and skillspector scan ./report-example --no-llm --format json --output report.json. Compare the benign HTML Output Rules heading with the existing prompt-extraction exception and the related cases in #37 and #453. Done means paired benign and malicious regressions pass while finding locations, risk accounting, and JSON, Markdown, and SARIF output remain correct in the complete CLI scan.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
security, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.