IntelliTect / IntelliTect/EssentialCSharp.Web
Security: No Content Safety Filtering or Output Guardrails on AI Responses
- Dominant language
- HTML
- Stars
- 8
- Forks
- 8
- Avg merge
- 17h 23m
- Merged PRs (30d)
- 61
Description
## Summary
The AI chat pipeline returns raw Azure OpenAI output directly to users with no output validation, PII detection, content safety filtering, or structured output enforcement. This means harmful, policy-violating, or sensitive content generated by the model would be passed straight through.
## Affected Code
**`EssentialCSharp.Web/Controllers/ChatController.cs`** — both `/api/chat/message` and `/api/chat/stream`:
```csharp
// Non-streaming: raw model text returned without any inspection
return Ok(new ChatMessageResponse { Response = response, ... });
// Streaming: raw delta text written directly to SSE stream
var eventData = JsonSerializer.Serialize(new { type = "text", data = text });
await Response.WriteAsync($"data: {eventData}\n\n", cancellationToken);
```
**`AIChatService.cs`** — `GetChatCompletionCore` returns `responseText` from the model with no post-processing.
There is also no content safety check on user **input** before it is sent to the model.
## Risk
**OWASP AI Agent Security — Risk #5: Output Validation & Guardrails / Risk #8: Sensitive Data Exposure**
- A jailbroken or misconfigured model could return harmful content, credentials from its training data, or PII.
- No defense against the model leaking content from its system prompt or injected context back to the user.
- No enforcement that outputs conform to expected shape (e.g., markdown response about C# topics only).
## Recommended Mitigations
1. **Integrate Azure AI Content Safety** for both input (before sending to model) and output (before returning to client):
```csharp
// In AIChatService or a middleware wrapper:
var inputSafety = await contentSafetyClient.AnalyzeTextAsync(prompt);
if (inputSafety.Value.CategoriesAnalysis.Any(c => c.Severity > 2))
return ("Content policy violation.", null);
```
2. **Apply a system prompt instruction** bounding output scope: "Only respond about C# and Essential C# book topics."
3. **Screen output for PII patterns** (emails, phone numbers, SSNs) before streaming/returning:
```csharp
private static readonly Regex PiiPattern = new(@"\b\d{3}-\d{2}-\d{4}\b|\b[\w.]+@[\w.]+\b", RegexOptions.Compiled);
```
4. **Set `max_output_tokens`** on the `ResponseCreationOptions` to enforce an output length cap and prevent runaway generation costs.
## References
- [OWASP AI Agent Security Cheat Sheet — §5 Output Validation & Guardrails](https://cheatsheetseries.owasp.org/cheatsheets/AI_Agent_Security_Cheat_Sheet.html#5-output-validation-guardrails)
- [Azure AI Content Safety](https://learn.microsoft.com/en-us/azure/ai-services/content-safety/overview)
- [OWASP LLM Top 10 — LLM02: Insecure Output Handling](https://owasp.org/www-project-top-10-for-large-language-model-applications/)
Contributor guide
Research direction
Start in EssentialCSharp.Web/Controllers/ChatController.cs by tracing the non-streaming and streaming response paths, then read AIChatService.cs and its GetChatCompletionCore method. Compare the input and output flows with the Azure AI Content Safety, PII screening, output-scope, and token-limit mitigations listed in the issue. Done means both chat endpoints and model inputs apply the agreed safety and output controls.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, csharp
- Domain
- ai, api, security
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100