IntelliTect / IntelliTect/EssentialCSharp.Web
Security: No Content Safety Filtering or Output Guardrails on AI Responses
- Langage dominant
- HTML
- Étoiles
- 8
- Forks
- 8
- Merge moyen
- 17 h 23 min
- PR mergées (30 j)
- 61
Description
## Summary
The AI chat pipeline returns raw Azure OpenAI output directly to users with no output validation, PII detection, content safety filtering, or structured output enforcement. This means harmful, policy-violating, or sensitive content generated by the model would be passed straight through.
## Affected Code
**`EssentialCSharp.Web/Controllers/ChatController.cs`** — both `/api/chat/message` and `/api/chat/stream`:
```csharp
// Non-streaming: raw model text returned without any inspection
return Ok(new ChatMessageResponse { Response = response, ... });
// Streaming: raw delta text written directly to SSE stream
var eventData = JsonSerializer.Serialize(new { type = "text", data = text });
await Response.WriteAsync($"data: {eventData}\n\n", cancellationToken);
```
**`AIChatService.cs`** — `GetChatCompletionCore` returns `responseText` from the model with no post-processing.
There is also no content safety check on user **input** before it is sent to the model.
## Risk
**OWASP AI Agent Security — Risk #5: Output Validation & Guardrails / Risk #8: Sensitive Data Exposure**
- A jailbroken or misconfigured model could return harmful content, credentials from its training data, or PII.
- No defense against the model leaking content from its system prompt or injected context back to the user.
- No enforcement that outputs conform to expected shape (e.g., markdown response about C# topics only).
## Recommended Mitigations
1. **Integrate Azure AI Content Safety** for both input (before sending to model) and output (before returning to client):
```csharp
// In AIChatService or a middleware wrapper:
var inputSafety = await contentSafetyClient.AnalyzeTextAsync(prompt);
if (inputSafety.Value.CategoriesAnalysis.Any(c => c.Severity > 2))
return ("Content policy violation.", null);
```
2. **Apply a system prompt instruction** bounding output scope: "Only respond about C# and Essential C# book topics."
3. **Screen output for PII patterns** (emails, phone numbers, SSNs) before streaming/returning:
```csharp
private static readonly Regex PiiPattern = new(@"\b\d{3}-\d{2}-\d{4}\b|\b[\w.]+@[\w.]+\b", RegexOptions.Compiled);
```
4. **Set `max_output_tokens`** on the `ResponseCreationOptions` to enforce an output length cap and prevent runaway generation costs.
## References
- [OWASP AI Agent Security Cheat Sheet — §5 Output Validation & Guardrails](https://cheatsheetseries.owasp.org/cheatsheets/AI_Agent_Security_Cheat_Sheet.html#5-output-validation-guardrails)
- [Azure AI Content Safety](https://learn.microsoft.com/en-us/azure/ai-services/content-safety/overview)
- [OWASP LLM Top 10 — LLM02: Insecure Output Handling](https://owasp.org/www-project-top-10-for-large-language-model-applications/)
Guide de contribution
Ouvrir le guide de contribution
Piste de recherche
Start in EssentialCSharp.Web/Controllers/ChatController.cs by tracing the non-streaming and streaming response paths, then read AIChatService.cs and its GetChatCompletionCore method. Compare the input and output flows with the Azure AI Content Safety, PII screening, output-scope, and token-limit mitigations listed in the issue. Done means both chat endpoints and model inputs apply the agreed safety and output controls.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Évaluation
- Stack technique
- azure, csharp
- Domaine
- ai, api, security
- Type d'issue
- Fonctionnalité
- Difficulté
- 5/5
- Temps estimé
- Plus d'une semaine
- Activité
- Calme
- Clarté
- Plutôt claire
- Accessibilité débutants
- 35/100