NVIDIA / NVIDIA/SkillSpector

[BUG] OpenAI-compatible provider + Gemini: report issues have null 'finding' and 'category' (explanation/severity OK)

Open
#304 2 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
17.9k
Forks
1.5k
Avg merge
5d 10h
Merged PRs (30d)
66

Description

Summary

When the active provider is openai pointed at Google's OpenAI-compatible endpoint (https://generativelanguage.googleapis.com/v1beta/openai/, model gemini-*-flash), LLM-stage issues in the JSON report come back with finding and category set to null, while explanation, id, severity, confidence, location, and remediation are populated correctly.

This doesn't lose the detection (severity/explanation are right) but it breaks any tooling that keys off finding / category, and it makes terminal/markdown output less readable.

Example

Scanning a test skill that exfiltrates ~/.config/gh/hosts.yml:

{
  "id": "SSD-3",
  "severity": "CRITICAL",
  "confidence": 1.0,
  "category": null,
  "finding": null,
  "explanation": "Step 3 attempts explicit credential exfiltration by POSTing ~/.config/gh/hosts.yml to an external endpoint (http://x.test/sync). ...",
  "remediation": "Completely delete step 3 ...",
  "location": {"file": "SKILL.md", "start_line": 17, "end_line": 20}
}

severity=CRITICAL with a correct explanation, but finding and category are null.

Likely cause

The structured-output schema mapping for OpenAI-compatible providers seems to leave these two fields unpopulated when the backend is Gemini (as opposed to stock OpenAI). It may be that the model returns them under different keys, or that with_structured_output on this endpoint doesn't enforce them. I haven't traced the exact spot.

Impact

Low severity — detection is intact — but it's a real interop rough edge for anyone using SkillSpector's growing OpenAI-compatible provider support with non-OpenAI backends (Gemini, and likely others). Worth a note either way: either populate finding/category, or document that they're best-effort on non-OpenAI backends.

Environment

SkillSpector (current main), SKILLSPECTOR_PROVIDER=openai, OPENAI_BASE_URL=https://generativelanguage.googleapis.com/v1beta/openai/, SKILLSPECTOR_MODEL=gemini-flash-latest / gemini-3.5-flash.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the report with SKILLSPECTOR_PROVIDER=openai, the Gemini OpenAI-compatible OPENAI_BASE_URL, and a Gemini flash model, then trace the structured-output schema mapping for this provider path. Done means finding and category are populated consistently, or the non-OpenAI limitation is documented; the payload names no files or tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, backend-api-design
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.