AI observability for chat

Borrow from https://www.braintrust.dev/docs/evaluation to come up with evaluation FW for CharBI demo.
Lean onto existing `nao/test` FW to pivot it to evaluating free-text responses: did it follow the rules, terminology compliance, semantic model, design principles.

10 strategic evals testing semantic understanding, disambiguation, root cause reasoning, and decision-support capabilities in a conversational analytics context.

## EVAL 1: Broad Intent Disambiguation – "Report on Reliability"

Category: Semantic disambiguation / Intent clarification

Problem Statement

User says: "Report on reliability"

Naive responses fail because "reliability" is ambiguous in EV charging:

Uptime (infrastructure availability)?
Success rate (successful charges)?
First-attempt success (charging completeness on first try)?
Failed visits (abandoned charges)?
Eval Criteria

✅ Pass: Model asks clarifying questions OR makes justified assumption and states it

E.g., "I'll look at first-attempt success rate and failure rates—the best indicators of whether your network is reliably completing charges for customers"
Maps "reliability" explicitly to specific metrics (visit_success_count, first_attempt_success_rate, failed_rate)
❌ Fail:

Returns a single metric without clarifying
Uses unrelated metrics (e.g., energy_per_visit_kwh_average)
Ignores the semantic breadth
Expected Behavior

Should recognize that "reliability" in EV charging context typically means customer success metrics (first-attempt success, failure rate) more than uptime, but a good response surfaces multiple interpretations and justifies the choice.

## EVAL 2: Vague Temporal Scope – "How Are We Doing Lately?"

Category: Temporal and scope disambiguation

Problem Statement

User asks: "How are we doing lately?"

Multiple ambiguities:

"We" = which business unit? All chargers? A region? A specific network operator?
"Doing" = profitability? Reliability? Growth? Customer satisfaction?
"Lately" = last 7 days? 30 days? Month-over-month? vs. baseline?
Eval Criteria

✅ Pass:

Identifies all three ambiguities
Asks at least 2 of the 3 for clarification
OR makes explicit assumptions: "I'll show you visit success metrics for the last 30 days across all charge points"
Provides metrics with a temporal window clearly stated
❌ Fail:

Ignores scope ("we" undefined)
Returns metrics without time context
Asks for clarification but then ignores the answer
Expected Behavior

A robust response acknowledges "lately" requires a time window definition, and either asks or states the assumption (e.g., "last 30 days" vs. "last week").

## EVAL 3: Industry Norms – "Is 87% Uptime Good?"

Category: Contextual judgment / domain knowledge

Problem Statement

User provides a metric: "Our uptime is 87%. Is that good?"

Requires domain knowledge because:

EV charging uptime benchmarks vary by network maturity, market, competitor baseline
87% could be excellent for emerging network, poor for mature infrastructure
Industry standard ≈ 95%+ for mature charging networks
Eval Criteria

✅ Pass:

Acknowledges that 87% is below typical EV charging industry standard (~95%)
Provides context: "For a commercial network, 87% suggests ~1 outage day/month, which impacts revenue & customer trust"
Suggests diagnostic next step: "Which charge points drive the gap? Are outages clustered?"
Offers action: "This typically prompts focus on preventive maintenance and vendor SLAs"
❌ Fail:

Says "looks fine" without context
Provides no industry benchmark
Doesn't connect 87% to business impact (cost, customer churn, energy revenue)
Expected Behavior

Grounded response with benchmarks, context, and diagnosis direction.

## EVAL 4: Failed Rate Interpretation – "What Does a 12% Failed Rate Mean?"

Category: Metric interpretation + action guidance

Problem Statement

User asks: "We have a 12% failed rate. What does that tell us?"

Requires interpreting failed_visits_count / total_visits ratio in context of:

Customer experience (1 in 8 visits fail completely)
Revenue impact (lost energy sales, customer frustration)
Operational issues (hardware failures? payment issues? network issues?)
Eval Criteria

✅ Pass:

Quantifies impact: "12% means roughly 1 in 8 visits don't result in energy transfer—potential loss of ~$XX in revenue/month (depending on avg payment)"
Breaks down root causes: "Is this due to first-attempt failures? Or failed visits after multiple troubled attempts?"
Asks diagnostic: "Are failures concentrated by charge_point_id, connector_id, or time period?"
Suggests action: "Investigate the top 3-5 failing chargers; review payment failure logs; check for firmware issues"
❌ Fail:

Just states the percentage as "below average" or "acceptable"
No business impact quantification
No root cause hypotheses
No suggested next steps
Expected Behavior

Interpret the metric as a business problem requiring investigation, not just a number.

## EVAL 5: Root Cause Analysis – Structured Decomposition

Category: Analytical depth / diagnostic reasoning

Problem Statement

User asks: "Why are our failures so high? Help me investigate."

Good root cause analysis requires decomposing failure into contributory factors:

By charger: Which charge_point_ids have high failure rates?
By attempt pattern: Is it first-attempt failures? Or troubled successes that eventually fail?
By time: Are failures clustered on specific days/hours (maintenance windows, high-volume periods)?
By connector: Specific connector_ids failing?
By stop reason: What reasons do failed attempts have? (Faulted status, no transaction, low energy)
Eval Criteria

✅ Pass:

Proposes a multi-dimensional analysis plan:
Segment failed_visits_count by charge_point_id (top 5 offenders)
Compare first_attempt_success_rate vs. troubled_success_rate (is it retry exhaustion?)
Compare visit_success_count by date_id (temporal pattern?)
Look at transaction_stop_reason (payment? hardware?)
Suggests specific queries or dashboard views
Prioritizes: "Start with charge_point_id—likely a 80/20 distribution"
❌ Fail:

Single-dimensional (only by charger, ignoring time/attempt pattern)
No hypothesis about root cause distribution
No prioritization (assumes all factors equally important)
Suggests investigation without narrowing scope
Expected Behavior

A structured diagnostic funnel that helps user explore systematically, not just "look at the data."

## EVAL 6: Comparative Judgment – "Is 6.2 Attempts/Visit Normal?"

Category: Contextual metrics / operational insight

Problem Statement

User states: "Our average attempts per visit is 6.2. Is that a problem?"

Misalignment signals:

Healthy system: 1.1–1.3 attempts/visit (most succeed first try, few troubled successes)
6.2 suggests rampant failure & retry (either poor network reliability OR poor payment/authentication)
Eval Criteria

✅ Pass:

Flags as anomalously high: "6.2 is well above healthy (typically 1.1–1.3)"
Connects to metrics: "This suggests either first_attempt_success_rate is very low OR many visits require multiple retries before failing"
Asks for decomposition: "What's your first_attempt_success_rate? And your troubled_success_rate?"
Suggests operational issue: "High attempts often indicate payment friction, authentication issues, or hardware intermittency"
Recommends investigation: "Analyze troubled_success_visits vs. failed_visits—where do attempts get exhausted?"
❌ Fail:

No comparative context (doesn't recognize as anomalous)
Doesn't connect to underlying success/failure metrics
No diagnosis or next step
Expected Behavior

Recognize the metric as a health signal, contextualize it, and guide investigation.

## EVAL 7: Feature Importance – "Why Focus on First-Attempt Success?"

Category: Metric prioritization / strategy articulation

Problem Statement

User asks: "You keep mentioning first-attempt success rate. Why is that more important than overall success rate?"

Requires explaining:

First-attempt success = customer friction reduction (better UX, faster charging)
Overall success = reliability (eventually charges work, even after retries)
Both matter, but first-attempt is a leading indicator of operational quality
High first-attempt success also correlates with lower cost-per-transaction (fewer retry infrastructure costs)
Eval Criteria

✅ Pass:

Distinguishes the two metrics: "Overall success tells you if your network works. First-attempt success tells you how smoothly it works."
Explains business value: "First-attempt success reduces customer frustration, improves user retention, and lowers transaction costs"
Suggests targeting: "If overall success is 90% but first-attempt is 60%, your network is unreliable—prioritize fixing infrastructure, not just adding retry logic"
Offers strategic framing: "First-attempt success is a leading indicator of operational health; overall success is a lagging indicator"
❌ Fail:

Treats them as equivalent
No business justification for prioritizing one
No strategic framing
Expected Behavior

Articulate why certain metrics deserve priority in decision-making.

## EVAL 8: Counterfactual Reasoning – "Should We Decommission This Charger?"

Category: Decision support / risk quantification

Problem Statement

User asks: "Charger X has a 4% first-attempt success rate. Should we decommission it?"

Requires counterfactual reasoning:

What's the cost of keeping it? Negative customer experience, potential revenue loss, maintenance costs
What's the cost of decommissioning? Lost revenue, reduced network availability, customer displacement to other chargers
What's the diagnosis? Is it a hardware failure (replace, don't decommission) or a network/location issue (decommission)?
What's the impact? How many visits per week? Average revenue per visit? Customer alternatives?
Eval Criteria

✅ Pass:

Asks for more context before recommending decommission:
"How many visits/week does this charger serve?"
"Is this a recent decline or long-standing?"
"Are failures concentrated in off-peak hours (maintenance window) or random?"
Suggests diagnosis first: "A 4% success rate suggests hardware failure or firmware issue—try a firmware update or diagnostics before decommissioning"
Quantifies impact: "If this charger does 20 visits/week at $X revenue, decommissioning costs $XX/month. Is the reputational cost worse?"
Recommends sequence: "Diagnose → attempt fix → monitor → decommission if unresolved"
❌ Fail:

Recommends decommission without diagnosis
Ignores counterfactual costs (lost revenue)
No consideration of customer impact
Treats it as a binary decision (decommission or keep)
Expected Behavior

Support a phased decision process: diagnose → remediate → monitor → decommission (if needed). Quantify trade-offs.

## EVAL 9: Temporal Trend Analysis – "Are We Getting Better or Worse?"

Category: Trend recognition / performance trajectory

Problem Statement

User asks: "Is our reliability improving or degrading over the last 3 months?"

Requires:

Temporal segmentation: Break into 3 monthly buckets
Key metrics tracked: first_attempt_success_rate, failed_rate, visits_count, uptime_average
Trend interpretation: Is the trend statistically significant or noise?
Causal reasoning: Did we make operational changes? Seasonal effects?
Eval Criteria

✅ Pass:

Segments data: "Month 1: 78% first-attempt success. Month 2: 81%. Month 3: 79%—slight improvement then regression"
Identifies volatility: "Trends are within 3 percentage points—not conclusive without more data"
Asks for context: "Did you deploy a firmware update? Seasonal variation? Growth in visit volume?"
Suggests causal analysis: "Compare trends by charge_point_id—if all improved, it's operational. If only some improved, it's localized issues"
Recommends action: "Drill into Month 3 regression—which chargers degraded? Review logs from that period"
❌ Fail:

Reports raw numbers without trend framing
No consideration of noise/significance
No causal hypotheses
No actionable next step
Expected Behavior

Recognize patterns, contextualize volatility, and recommend investigation direction.

## EVAL 10: Cross-Metric Coherence – Spot Inconsistencies

Category: Data quality / sanity check

Problem Statement

User reports: "First-attempt success rate is 92%, but overall success rate is also 92%. Troubled success rate is 8%. That doesn't add up."

Requires validating metric logic:

Correct math: first_attempt_success + troubled_success + failed = 100%
If first-attempt = 92% and troubled = 8%, then failed = 0% (makes sense)
But if overall success = 92%, then failed should also = 8% (not 0%)
Inconsistency signals potential issues:
Metric definition misalignment
Data pipeline bug
User misinterpretation of metric definitions
Eval Criteria

✅ Pass:

Catches the inconsistency: "Your numbers don't reconcile. If first-attempt + troubled = 100% of successful visits, then overall success should equal first-attempt + troubled"
Clarifies definitions: "Let me confirm—are these percentages of all visits, or percentages within the success population?"
Suggests root cause: "This might indicate a data pipeline issue or a misalignment in how troubled_success is calculated"
Recommends verification: "Compare first_attempt_success_count + troubled_success_count + failed_visits_count—they should sum to visits_count"
❌ Fail:

Accepts inconsistency without noting it
Assumes user's numbers are correct
Doesn't validate metric logic
Expected Behavior

Act as a sanity-check partner; catch logical inconsistencies in user claims.

Summary Table

Implementation Considerations

For Each Eval:

Create realistic test cases with specific data snapshots (e.g., "first-attempt success = 62%, troubled success = 18%, failed = 20%")
Define scoring rubric: Does the response mention X? Ask clarifying question Y? Quantify impact Z?
Test both LLM and retrieval: Does the chatBI retrieve the right semantic model entities?
Edge cases: What if data is missing? What if the user's assumption is wrong?
Eval Difficulty Progression:

Easy (1): Sanity checks
Medium (5): Semantic disambiguation, metric interpretation, comparative judgment
Hard (3): Deep diagnosis, strategic prioritization, counterfactual reasoning
Suggested Test Data Patterns:

Healthy system: ~92% first-attempt success, ~5% troubled success, ~3% failed, 1.08 attempts/visit, 95%+ uptime
Degrading system: 60% first-attempt, 25% troubled, 15% failed, 3.2 attempts/visit, 82% uptime
Anomalous charger: One charger with 4% success vs. network avg 88%
Temporal pattern: Success trending 88% → 82% → 76% over 3 months