atxtechbro / atxtechbro/dotfiles

Track MCP tool rejection/denial rates for AI agent behavior analysis

Open
#462 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Shell
Stars
27
Forks
2
PR merge metrics
No merged PRs in 30d

Description

## Problem Statement

When Amazon Q CLI prompts for tool confirmation and the user responds with anything other than 'y' (yes), it's unclear whether this rejection is logged in our MCP tool logging system. This creates a blind spot in understanding AI agent behavior and user trust patterns.

## Example Scenario

```
🛠️ Using tool: git_add from mcp server git

● Running git_add with the param:
⋮ {
⋮ "arguments": {
⋮ "files": [
⋮ "."
⋮ ],
⋮ "repo_path": "/Users/morgan.joyce/ppv/pillars/dotfiles"
⋮ },
⋮ "name": "git_add"
⋮ }
Allow this action? Use 't' to trust (always allow) this tool for the session. [y/n/t]:

> dont willy nilly dd all everything should have already been commited maybe check git status
```

In this case, the AI agent suggested adding all files (`.`) without first checking git status, which the user rightfully rejected.

## Questions to Investigate

1. **Current Logging Behavior**: When a user responds with 'n' or any non-'y' response, is this logged in our MCP tool logging system (`~/mcp-tool-calls.log`)?

2. **Rejection Tracking**: If rejections aren't currently logged, how can we implement logging for:
- Tool rejection events
- Rejection reasons (user's response text)
- Context that led to the rejection

3. **Metrics Value**: What insights could we gain from tracking rejection rates:
- Which tools are most frequently rejected?
- What patterns lead to rejections?
- How can this improve AI agent behavior?

## Implementation Considerations

### Current MCP Logging System
Our MCP servers use `logging_utils.py` with functions like:
- `log_tool_success()` - logs successful tool executions
- `log_tool_error()` - logs tool execution errors

### Potential Enhancement
We may need to add:
- `log_tool_rejected()` - logs when user denies tool execution
- Enhanced logging to capture user feedback/reasoning

### Technical Challenges
- **Q CLI Integration**: The tool confirmation happens at the Q CLI level, not within our MCP server
- **Logging Location**: Rejections might need to be logged differently than successful executions
- **Context Preservation**: Capturing the user's reasoning for rejection

## Use Cases

### AI Agent Behavior Analysis
- Track which tool suggestions are frequently rejected
- Identify patterns in poor AI decision-making
- Improve agent prompting to reduce unnecessary tool calls

### User Trust Metrics
- Measure user confidence in AI suggestions
- Identify tools that need better context or validation
- Track improvement in AI behavior over time

### Development Insights
- Understand which MCP tools need better safeguards
- Identify when AI agents make assumptions without proper checks
- Guide development of smarter tool usage patterns

## Research Tasks

1. **Test Current Behavior**:
- Trigger a tool confirmation prompt
- Respond with 'n' or explanatory text
- Check if anything appears in `~/mcp-tool-calls.log`

2. **Investigate Q CLI Logging**:
- Determine where/how Q CLI handles tool rejections
- Identify if there are existing hooks for rejection logging

3. **Design Logging Enhancement**:
- Propose mechanism for capturing rejections
- Consider integration points with existing MCP logging
- Design schema for rejection events

## Expected Outcomes

- Understanding of current rejection logging behavior
- Proposal for enhanced rejection tracking
- Implementation plan for rejection rate analytics
- Improved AI agent behavior through feedback loops

## Related Work

- Our existing MCP tool logging system (`mcp/README.md#adding-tool-level-logging-to-mcp-servers`)
- MCP protocol specifications for tool execution
- Amazon Q CLI tool permission and trust management

This enhancement would provide valuable insights into AI agent behavior and help build more trustworthy automated systems.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with mcp/README.md#adding-tool-level-logging-to-mcp-servers and logging_utils.py, then trigger a Q CLI tool confirmation and inspect ~/mcp-tool-calls.log after a non-'y' response. Determine whether Q CLI exposes rejection details to the MCP servers. Done means documenting current behavior and an agreed design for rejection events, context, and rejection-rate analytics.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, shell
Domain
ai, cli, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.