NVIDIA-NeMo / NVIDIA-NeMo/Guardrails
[Security] LLM Prompt Injection Not Prevented - Jailbreak Vulnerability
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.2k
- Forks
- 843
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 25
Description
Description
Guardrails doesn't prevent prompt injection attacks. Malicious prompts override safety guidelines and generate harmful content.
Expected Behavior
Implement prompt structure validation. Detect and reject injection attempts.
Actual Behavior
File: src/rails/guardrails.py
No injection detection: llm.generate(user_prompt)
GSSoC Points Estimate: Level 3 (Security/LLM)
Suggested Labels
- gssoc:approved
- type:bug
- severity:critical
- area:security
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with src/rails/guardrails.py and inspect the llm.generate(user_prompt) path. Define how prompt structure validation should detect and reject injection attempts, then verify that malicious prompts no longer override the safety guidelines.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, security
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100