IntelliTect / IntelliTect/EssentialCSharp.Web

Security: No Content Safety Filtering or Output Guardrails on AI Responses

Open
#1,065 0 comments 0 reactions 0 assignees View on GitHub
ai-agent security
Dominant language
HTML
Stars
8
Forks
8
Avg merge
17h 23m
Merged PRs (30d)
61

Description

## Summary
The AI chat pipeline returns raw Azure OpenAI output directly to users with no output validation, PII detection, content safety filtering, or structured output enforcement. This means harmful, policy-violating, or sensitive content generated by the model would be passed straight through.

## Affected Code
**`EssentialCSharp.Web/Controllers/ChatController.cs`** — both `/api/chat/message` and `/api/chat/stream`:

```csharp
// Non-streaming: raw model text returned without any inspection
return Ok(new ChatMessageResponse { Response = response, ... });

// Streaming: raw delta text written directly to SSE stream
var eventData = JsonSerializer.Serialize(new { type = "text", data = text });
await Response.WriteAsync($"data: {eventData}\n\n", cancellationToken);
```

**`AIChatService.cs`** — `GetChatCompletionCore` returns `responseText` from the model with no post-processing.

There is also no content safety check on user **input** before it is sent to the model.

## Risk
**OWASP AI Agent Security — Risk #5: Output Validation & Guardrails / Risk #8: Sensitive Data Exposure**

- A jailbroken or misconfigured model could return harmful content, credentials from its training data, or PII.
- No defense against the model leaking content from its system prompt or injected context back to the user.
- No enforcement that outputs conform to expected shape (e.g., markdown response about C# topics only).

## Recommended Mitigations

1. **Integrate Azure AI Content Safety** for both input (before sending to model) and output (before returning to client):
```csharp
// In AIChatService or a middleware wrapper:
var inputSafety = await contentSafetyClient.AnalyzeTextAsync(prompt);
if (inputSafety.Value.CategoriesAnalysis.Any(c => c.Severity > 2))
return ("Content policy violation.", null);
```
2. **Apply a system prompt instruction** bounding output scope: "Only respond about C# and Essential C# book topics."
3. **Screen output for PII patterns** (emails, phone numbers, SSNs) before streaming/returning:
```csharp
private static readonly Regex PiiPattern = new(@"\b\d{3}-\d{2}-\d{4}\b|\b[\w.]+@[\w.]+\b", RegexOptions.Compiled);
```
4. **Set `max_output_tokens`** on the `ResponseCreationOptions` to enforce an output length cap and prevent runaway generation costs.

## References
- [OWASP AI Agent Security Cheat Sheet — §5 Output Validation & Guardrails](https://cheatsheetseries.owasp.org/cheatsheets/AI_Agent_Security_Cheat_Sheet.html#5-output-validation-guardrails)
- [Azure AI Content Safety](https://learn.microsoft.com/en-us/azure/ai-services/content-safety/overview)
- [OWASP LLM Top 10 — LLM02: Insecure Output Handling](https://owasp.org/www-project-top-10-for-large-language-model-applications/)

Contributor guide

Open the contributing guide

Research direction

Start in EssentialCSharp.Web/Controllers/ChatController.cs by tracing the non-streaming and streaming response paths, then read AIChatService.cs and its GetChatCompletionCore method. Compare the input and output flows with the Azure AI Content Safety, PII screening, output-scope, and token-limit mitigations listed in the issue. Done means both chat endpoints and model inputs apply the agreed safety and output controls.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, csharp
Domain
ai, api, security
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.