NVIDIA / NVIDIA/SkillSpector

[Feature] Expose the file surface a finding landed on (code / instructions / docs / tests / config)

Open
#326 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
17.9k
Forks
1.5k
Avg merge
5d 10h
Merged PRs (30d)
66

Description

Summary

Findings do not say what kind of file surface they landed on, and triage is much harder without it. This is a request to expose that information, not to filter on it — filtering is contentious for good reasons (see below), while the data itself is useful either way.

The measurement

Scanning 65 real skill/plugin units — 4415 findings, every one triaged by hand — and classifying each finding by the surface its line belongs to:

rule findings code instructions tests docs comments config
RP1 894 262 94 64 381 72 21
PE3 814 296 22 211 115 156 14
AST4 417 238 0 178 0 1 0
E1 390 69 7 262 36 7 9
RA2 235 88 24 24 44 48 7
AS3 207 7 44 33 48 6 69
MP2 194 48 20 24 87 15 0
EA3 175 103 1 8 62 1 0

Totals across all 53 rules: code 1442 · tests 981 · docs 980 · comments 412 · instructions (SKILL.md) 325 · config 275.

instructions is kept separate from docs on purpose: SKILL.md is not prose about a command, it is what the agent executes — which is exactly why _is_documentation_context already excludes it. Counting it as documentation overstates the "non-executable" share, and I made that mistake before correcting it.

Why exposing, and not filtering

Two of these columns look like obvious suppression candidates. They are not:

  • tests (981 findings). A fixture containing an attack string is expected — but a directory named tests/ is also exactly where a payload would hide. Exempting it trades a real detection for a cosmetic gain.
  • config — and the reason the aggregate is misleading. SC1 has 100% of its findings outside executable code, entirely in config files. For a dependency-pinning rule that is the correct place. "Not code" does not mean "noise", and the ratio only means something per rule.

Both arguments point the same way: the classification is genuinely useful for whoever reads the report, and genuinely dangerous as an automatic filter. So ship the data and let the consumer decide.

What this would look like

A normalised field on each finding — surface: code | instructions | docs | tests | config — derived from the path (and, for the comment case, from the matched line). The runner already computes file_type and _infer_file_type, so most of the input exists.

Concretely it would let a consumer:

  • sort a 400-finding report by surface and read the 238 code ones first;
  • write a suppression policy locally, taking on the risk explicitly, instead of the scanner taking it on their behalf;
  • diff two scans by surface, which is how one notices that a rule started matching documentation after a version bump — how the PE3 change in 2.4/2.5 became visible here at all.

I have not opened a PR because this touches the report schema, which is a public contract and yours to design. If the shape is welcome I am happy to implement it; if the answer is "the consumer can derive this from location.file", that is a fair answer too — it is what I ended up doing, and the reason I can post the table above.

Context

From the same corpus that produced #302, #319, #321, #322 and #323. Zero true positives in 4415 findings, which is a statement about this corpus (first-party and well-known marketplaces), not about the scanner.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the runner’s existing file_type and _infer_file_type logic, then read the report schema and location.file handling. Work out how the requested surface values should be represented, including matched-line comments, without adding filtering. Done means each finding exposes a consistent surface value that consumers can use in reports and schema-based integrations.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
security, tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.