IntelliTect / IntelliTect/EssentialCSharp.Web
Security: No Content Safety Filtering or Output Guardrails on AI Responses
- Vorherrschende Sprache
- HTML
- Sterne
- 8
- Forks
- 8
- Ø Merge
- 17 Std. 23 Min.
- Gemergte PRs (30 T.)
- 61
Beschreibung
## Summary
The AI chat pipeline returns raw Azure OpenAI output directly to users with no output validation, PII detection, content safety filtering, or structured output enforcement. This means harmful, policy-violating, or sensitive content generated by the model would be passed straight through.
## Affected Code
**`EssentialCSharp.Web/Controllers/ChatController.cs`** — both `/api/chat/message` and `/api/chat/stream`:
```csharp
// Non-streaming: raw model text returned without any inspection
return Ok(new ChatMessageResponse { Response = response, ... });
// Streaming: raw delta text written directly to SSE stream
var eventData = JsonSerializer.Serialize(new { type = "text", data = text });
await Response.WriteAsync($"data: {eventData}\n\n", cancellationToken);
```
**`AIChatService.cs`** — `GetChatCompletionCore` returns `responseText` from the model with no post-processing.
There is also no content safety check on user **input** before it is sent to the model.
## Risk
**OWASP AI Agent Security — Risk #5: Output Validation & Guardrails / Risk #8: Sensitive Data Exposure**
- A jailbroken or misconfigured model could return harmful content, credentials from its training data, or PII.
- No defense against the model leaking content from its system prompt or injected context back to the user.
- No enforcement that outputs conform to expected shape (e.g., markdown response about C# topics only).
## Recommended Mitigations
1. **Integrate Azure AI Content Safety** for both input (before sending to model) and output (before returning to client):
```csharp
// In AIChatService or a middleware wrapper:
var inputSafety = await contentSafetyClient.AnalyzeTextAsync(prompt);
if (inputSafety.Value.CategoriesAnalysis.Any(c => c.Severity > 2))
return ("Content policy violation.", null);
```
2. **Apply a system prompt instruction** bounding output scope: "Only respond about C# and Essential C# book topics."
3. **Screen output for PII patterns** (emails, phone numbers, SSNs) before streaming/returning:
```csharp
private static readonly Regex PiiPattern = new(@"\b\d{3}-\d{2}-\d{4}\b|\b[\w.]+@[\w.]+\b", RegexOptions.Compiled);
```
4. **Set `max_output_tokens`** on the `ResponseCreationOptions` to enforce an output length cap and prevent runaway generation costs.
## References
- [OWASP AI Agent Security Cheat Sheet — §5 Output Validation & Guardrails](https://cheatsheetseries.owasp.org/cheatsheets/AI_Agent_Security_Cheat_Sheet.html#5-output-validation-guardrails)
- [Azure AI Content Safety](https://learn.microsoft.com/en-us/azure/ai-services/content-safety/overview)
- [OWASP LLM Top 10 — LLM02: Insecure Output Handling](https://owasp.org/www-project-top-10-for-large-language-model-applications/)
Beitragsleitfaden
Bewertung
Dieses Issue wurde noch nicht bewertet.