Static v2.11.1: distinguish inert security-test strings and taxonomy labels from executable behavior in findings

We integrated SkillSpector as an attributed independent evidence provider in
AI SAFE². Thank you for exposing coverage and degraded-analysis status explicitly;
those fields are important to our downstream review process.

Environment and scope

• SkillSpector v2.11.1, pinned source revision
  704bc9544260c2f41222dc0f92982521709496ab.
• Windows; Python 3.13.14; scan TARGET --format json --no-llm.
• Scan date: 2026-09-09. No target skill/test code executed by us.
• Model credentials were not forwarded. Static mode is not a claim of zero
  network access: supply-chain lookups can contact OSV.
• Public target: AI SAFE²'s tracked skills/ tree at
  6455eec76caec259ba3f9b902a4ab24f27b34263.
• This is a version-pinned observation, not a claim that every subsequent
  upstream revision is affected.

Observed result

The tracked-source snapshot produced 50 issues and a risk assessment of
100 / CRITICAL / DO_NOT_INSTALL. A separate development checkout also produced
50 issues. Our clean and inert-hostile controls produced zero and seven issues,
respectively. These counts are not a calibrated precision or recall benchmark.

Several explanations appear to infer executable behavior from inert content:

1. Prompt-injection strings in sanitizer regression tests are test inputs, with
   assertions about sanitization. A lexical match should not itself establish
   that the package follows those instructions.
2. The self_modification risk-taxonomy key describes a risk being assessed;
   it is not an operation that modifies the program or its policy.
3. The similarity between the package names uvicorn and gunicorn is insufficient
   on its own to establish typosquatting.
4. An environment-derived health-check URL is not itself a credential. Any
   exfiltration explanation should identify the actual sensitive source and sink.

Important caveat: some smoke tests send bearer tokens to a configured URL.
Their transport/destination validation merits separate review. We are not asking
for all test-path findings to be suppressed or claiming every finding is false.
Coverage warnings, degraded OSV checks, and unresolved references are genuine
limitations and should remain visible.

Reproduction

Recommend using a clean, disposable directory and isolated environment. Install the pinned scanner from its official source, not an unpinned dependency set. The source pin
does not itself pin all transitive dependencies.

git clone https://github.com/CyberStrategyInstitute/ai-safe2-framework.git safe2-repro
git -C safe2-repro archive --format=zip --output=../safe2-skills.zip 6455eec76caec259ba3f9b902a4ab24f27b34263 skills

Extract that public-source archive into a new specimen directory using your
platform's archive utility. Scan only; do not install or execute specimen code:

skillspector --version
skillspector scan specimen/skills --format json --no-llm --output report.json

The historical count is environment-dependent, particularly for network checks;
the useful reproduction target is the particular finding explanation and source
context, not an invariant total of 50. A fresh minimal specimen and current-main
comparison have not yet been run for this upstream submission.

Requested improvement / questions

• Distinguish lexical pattern matches, data-flow evidence, and confirmed
  executable behavior in the explanation and confidence semantics.
• Consider syntax/data-flow context for inert strings and taxonomy definitions.
• Preserve raw findings and coverage independently from reviewed dispositions.
• Add paired regression cases: inert test input versus active instruction/code,
  including malicious content placed under a test-like path.
• Do not exempt tests/, private IPs, or loopback wholesale: those are not trust
  boundaries. Baselines should be reviewer-controlled and narrowly scoped.
• Is this already covered by an existing issue, recommended baseline workflow,
  or a newer analyzer revision? We can help supply focused specimens.

Our integration preserves SkillSpector's original score and recommendation. We
do not rewrite those into approval or claim NVIDIA endorsement or conformance.

Public supporting analysis

AI SAFE² second-pass findings and limitations

Related reports

Related: #37 (text-content false positives), #314 (security-test patterns flagged), and #103 (earlier static documentation findings). This report supplies a pinned v2.11.1 AI SAFE2 specimen and distinguishes specific explanation/precision concerns from genuine coverage limitations. Please consolidate if an existing issue is the preferred tracking location.

Start by reproducing the pinned v2.11.1 scan with the provided archive and skillspector scan specimen/skills --format json --no-llm --output report.json. Review related issues #37, #314, and #103 before locating the analyzer and explanation logic. Done should distinguish lexical matches, data-flow evidence, and executable behavior while retaining raw findings and coverage, with paired inert-versus-active regression cases.