microsoft / microsoft/agent-framework

.NET: Feature Request: Built-in Security & Validation Middleware for AI Function Tools

Open
#2,254 5 comments 0 reactions 1 assignee Claimed by @stephentoub View on GitHub
.NET
Dominant language
Python
Stars
13.6k
Forks
2.3k
Avg merge
2d 45m
Merged PRs (30d)
358

Description

### Summary
The current Agent Framework provides a clean mechanism for defining and registering AIFunction tools using `AIFunctionFactory.Create().`
However, there is no built-in security or validation pipeline to intercept, inspect, or reject tool inputs before execution.

This creates real security risks when exposing tools that interact with:

- File systems
- HTTP requests
- SQL databases
- External services
- Any sensitive action the model may trigger

I am proposing a Tool Security Middleware Layer similar to middlewares in ASP.NET Core or OpenAI Function Call Interceptors.

### **Problem**

When an LLM runs a tool, the input arguments are passed directly to the underlying .NET function without:

- Sanitization
- Validation
- Policy checks
- Security restrictions
- Usage auditing
- Role-based access

Therefore, if a user prompt successfully triggers a tool call, the agent may execute unsafe behavior, e.g.:
**1. FileSystemTool**
A user could attempt:
`"Delete everything under C:\ or read system32 files"`

**2. HttpTool (SSRF)**
`"Call http://127.0.0.1:5000/admin/shutdown"`

**3. SQL Tool (SQL Injection)**
`"Run: SELECT * FROM Users; DROP TABLE Users"`

**4. Prompt Injection**
If an LLM is manipulated into calling a tool unintentionally, the framework cannot block it.

### **Proposed Solution**

Introduce a Tool Execution Middleware Pipeline:

- Similar to ASP.NET Core's middleware model
- Intercepts before tool execution
- Allows validating, modifying, or rejecting input
- Allows logging/auditing
- Allows policy-based access control

**Proposed API**
Validation Hook on AIFunctionFactory
```
AIFunctionFactory.Create(
method: SqlTool.Query,
validator: SqlValidator.ValidateQuery);
```

Where validator returns:

- null ⇒ valid
- string ⇒ error message

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.