spring-projects / spring-projects/spring-ai

Feature Request: Add tool success/failure metrics for production monitoring

Open
#3,853 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

status: waiting-for-triage
Dominant language
Java
Stars
9.5k
Forks
2.9k
Avg merge
1d 10h
Merged PRs (30d)
5

Description

Expected Behavior

Spring AI should provide success/failure metrics for individual tool calls to enable production monitoring and reliability tracking.

Proposed Metrics
  1. Tool Result Counter: spring.ai.tool.result
  • Tags: tool.name, success (true/false)
  • Tracks individual tool execution outcomes
  1. Tool Error Counter: spring.ai.tool.error
  • Tags: tool.name, error.type
  • Tracks tool failures by error type
Code Example
# Configuration (optional, enabled by default)
spring:
 ai:
   tool:
     metrics:
       enabled: true
       include-error-details: false 


# View tool success metrics
curl http://localhost:8080/actuator/metrics/spring.ai.tool.result

# Expected response:
{
  "name": "spring.ai.tool.result",
  "measurements": [
    { "statistic": "COUNT", "value": 45 }
  ],
  "availableTags": [
    { "tag": "tool.name", "values": ["getCurrentWeather", "getRecommendation"] },
    { "tag": "success", "values": ["true", "false"] }
  ]
}

# Filter by specific tool and outcome
curl "http://localhost:8080/actuator/metrics/spring.ai.tool.result?tag=tool.name:getCurrentWeather&tag=success:false" 

Implementation Approach

Add a new ToolMetricsObservationHandler that extends the existing observation system:

@Component
@ConditionalOnClass({ObservationRegistry.class, MeterRegistry.class})
public class ToolMetricsObservationHandler implements ObservationHandler<Observation.Context> {
    
    @Override
    public void onStop(Observation.Context context) {
        if (context instanceof ToolCallingObservationContext toolContext) {
            String toolName = toolContext.getToolDefinition().name();
            boolean isSuccess = !toolContext.hasError();
            
            Counter.builder("spring.ai.tool.result")
                .tag("tool.name", toolName)
                .tag("success", String.valueOf(isSuccess))
                .register(meterRegistry)
                .increment();
        }
    }
}

Current Behavior

Spring AI currently provides basic spring.ai.tool timer metrics through the existing observation system:

✅ Tool execution time (COUNT, TOTAL_TIME, MAX)
✅ Basic tool call tracking
❌ Success/failure breakdown
❌ Error rate monitoring
❌ Tool reliability insights

The existing metrics are generated automatically by Micrometer's DefaultMeterObservationHandler, but lack domain-specific insights needed for production monitoring.

Context

How has this issue affected you?
When running AI applications in production, it's critical to monitor tool reliability. Currently, there's no way to:

Identify which tools are failing frequently
Set up alerts for tool failure rates
Track tool performance degradation over time
Monitor SLA compliance for tool-dependent services

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading Spring AI’s existing tool observation system, including ToolCallingObservationContext and the DefaultMeterObservationHandler path, to determine where stopped tool outcomes and the MeterRegistry are available. Done means production monitoring exposes the proposed result and error metrics with the specified tags, configuration behavior, and actuator responses.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spring
Domain
backend, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.