IntelliTect / IntelliTect/EssentialCSharp.Web

Security: No Content Safety Filtering or Output Guardrails on AI Responses

Open
#1,065 0 comments 0 reactions 0 assignees View on GitHub
ai-agent security
Dominant language
HTML
Stars
8
Forks
8
Avg merge
17h 23m
Merged PRs (30d)
61

Description

## Summary
The AI chat pipeline returns raw Azure OpenAI output directly to users with no output validation, PII detection, content safety filtering, or structured output enforcement. This means harmful, policy-violating, or sensitive content generated by the model would be passed straight through.

## Affected Code
**`EssentialCSharp.Web/Controllers/ChatController.cs`** — both `/api/chat/message` and `/api/chat/stream`:

```csharp
// Non-streaming: raw model text returned without any inspection
return Ok(new ChatMessageResponse { Response = response, ... });

// Streaming: raw delta text written directly to SSE stream
var eventData = JsonSerializer.Serialize(new { type = "text", data = text });
await Response.WriteAsync($"data: {eventData}\n\n", cancellationToken);
```

**`AIChatService.cs`** — `GetChatCompletionCore` returns `responseText` from the model with no post-processing.

There is also no content safety check on user **input** before it is sent to the model.

## Risk
**OWASP AI Agent Security — Risk #5: Output Validation & Guardrails / Risk #8: Sensitive Data Exposure**

- A jailbroken or misconfigured model could return harmful content, credentials from its training data, or PII.
- No defense against the model leaking content from its system prompt or injected context back to the user.
- No enforcement that outputs conform to expected shape (e.g., markdown response about C# topics only).

## Recommended Mitigations

1. **Integrate Azure AI Content Safety** for both input (before sending to model) and output (before returning to client):
```csharp
// In AIChatService or a middleware wrapper:
var inputSafety = await contentSafetyClient.AnalyzeTextAsync(prompt);
if (inputSafety.Value.CategoriesAnalysis.Any(c => c.Severity > 2))
return ("Content policy violation.", null);
```
2. **Apply a system prompt instruction** bounding output scope: "Only respond about C# and Essential C# book topics."
3. **Screen output for PII patterns** (emails, phone numbers, SSNs) before streaming/returning:
```csharp
private static readonly Regex PiiPattern = new(@"\b\d{3}-\d{2}-\d{4}\b|\b[\w.]+@[\w.]+\b", RegexOptions.Compiled);
```
4. **Set `max_output_tokens`** on the `ResponseCreationOptions` to enforce an output length cap and prevent runaway generation costs.

## References
- [OWASP AI Agent Security Cheat Sheet — §5 Output Validation & Guardrails](https://cheatsheetseries.owasp.org/cheatsheets/AI_Agent_Security_Cheat_Sheet.html#5-output-validation-guardrails)
- [Azure AI Content Safety](https://learn.microsoft.com/en-us/azure/ai-services/content-safety/overview)
- [OWASP LLM Top 10 — LLM02: Insecure Output Handling](https://owasp.org/www-project-top-10-for-large-language-model-applications/)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.