forcedotcom / forcedotcom/cli

Testing Center custom `string_comparison` evaluation errors on the JSONPath the server itself generates from `$.generatedData.outcome`

Open
#3,636 2 comments 0 reactions 0 assignees View on GitHub
area:afdx investigating owned by another team validated
Dominant language
No language data
Stars
571
Forks
80
Avg merge
2d 15h
Merged PRs (30d)
3

Description

CLI Version:
@salesforce/cli/2.149.9

Architecture:
win32-arm64

Node Version:
node-v24.14.1

Plugin Version:
@oclif/plugin-autocomplete 3.2.56 (core)
@oclif/plugin-commands 4.1.63 (core)
@oclif/plugin-help 6.2.58 (core)
@oclif/plugin-not-found 3.2.93 (core)
@oclif/plugin-plugins 5.4.87 (core)
@oclif/plugin-search 1.2.54 (core)
@oclif/plugin-update 4.7.59 (core)
@oclif/plugin-version 2.2.57 (core)
@oclif/plugin-warn-if-update-available 3.1.73 (core)
@oclif/plugin-which 3.2.61 (core)
@salesforce/cli 2.149.9 (core)
agent 2.0.5 (user)
apex 4.1.0 (core)
api 2.0.9 (core)
auth 5.0.6 (core)
code-analyzer 5.12.0 (user)
data 5.1.5 (core)
deploy-retrieve 4.1.2 (core)
info 4.0.9 (core)
limits 4.0.3 (core)
marketplace 2.0.5 (core)
org 6.0.9 (core)
packaging 3.0.5 (core)
schema 4.0.5 (core)
settings 3.0.5 (core)
sobject 2.0.5 (core)
telemetry 4.0.5 (core)
templates 57.0.9 (core)
trust 4.0.9 (core)
user 5.0.1 (core)

OS and Version:
Windows_NT 10.0.26200

Shell:
powershell

### Summary

A custom evaluation asserting on the agent's response text is unusable: the
server rewrites the documented `$.generatedData.outcome` reference into a
filter-expression JSONPath and then fails to parse its own rewrite.

Test case YAML (per the documented shape):

```yaml
customEvaluations:
- label: "response contains estimat"
name: string_comparison
parameters:
- name: operator
value: contains
isReference: false
- name: actual
value: "$.generatedData.outcome"
isReference: true
- name: expected
value: "estimat"
isReference: false
```

### Actual result

The evaluation returns `status: ERROR` on every case:

```
"errorMessage": "Error parsing JSONPath
'$.outputs[?(@.type == 'general.echo')].payload.planner_response.lastExecution.outcome'"
```

i.e. the server maps `$.generatedData.outcome` to a path containing a
filter expression `[?(@.type == '...')]`, and the JSONPath implementation
evaluating it rejects filter syntax. The author's input is well-formed; the
failing path is generated internally.

### Expected result

Either the generated path parses, or `$.generatedData.outcome` maps to a
filter-free path, or the reference is rejected at test-create time with a
clear message. An ERROR status at run time on the documented reference makes
deterministic response-text assertions impossible in Testing Center.

### Why it matters

The LLM-judged `expectedOutcome` is lenient (see reproduction context in the
related issues: an agent whose instruction clause was removed still passed a
judge rubric that named the missing behavior). `string_comparison` is the
platform's only deterministic assertion over response text — precisely the
tool needed when the judge is too soft — and it errors as documented.

### Environment

- sf CLI:
- Org: Agentforce Developer Edition (`orgfarm-*.develop.my.salesforce.com`)

Contributor guide

Open the contributing guide

Research direction

No source files or tests are named. Start by reproducing the custom string_comparison evaluation with the documented $.generatedData.outcome reference and inspect how the server rewrites and parses that path. Done means the reference evaluates successfully, maps to a parseable path, or is rejected at creation with a clear error.

Written by the indexing model from the issue text.

Assessment

Domain
cli, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.