Testing Center custom `string_comparison` evaluation errors on the JSONPath the server itself generates from `$.generatedData.outcome`
- Dominant language
- No language data
- Stars
- 571
- Forks
- 80
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 3
Description
CLI Version:
@salesforce/cli/2.149.9
Architecture:
win32-arm64
Node Version:
node-v24.14.1
Plugin Version:
@oclif/plugin-autocomplete 3.2.56 (core)
@oclif/plugin-commands 4.1.63 (core)
@oclif/plugin-help 6.2.58 (core)
@oclif/plugin-not-found 3.2.93 (core)
@oclif/plugin-plugins 5.4.87 (core)
@oclif/plugin-search 1.2.54 (core)
@oclif/plugin-update 4.7.59 (core)
@oclif/plugin-version 2.2.57 (core)
@oclif/plugin-warn-if-update-available 3.1.73 (core)
@oclif/plugin-which 3.2.61 (core)
@salesforce/cli 2.149.9 (core)
agent 2.0.5 (user)
apex 4.1.0 (core)
api 2.0.9 (core)
auth 5.0.6 (core)
code-analyzer 5.12.0 (user)
data 5.1.5 (core)
deploy-retrieve 4.1.2 (core)
info 4.0.9 (core)
limits 4.0.3 (core)
marketplace 2.0.5 (core)
org 6.0.9 (core)
packaging 3.0.5 (core)
schema 4.0.5 (core)
settings 3.0.5 (core)
sobject 2.0.5 (core)
telemetry 4.0.5 (core)
templates 57.0.9 (core)
trust 4.0.9 (core)
user 5.0.1 (core)
OS and Version:
Windows_NT 10.0.26200
Shell:
powershell
### Summary
A custom evaluation asserting on the agent's response text is unusable: the
server rewrites the documented `$.generatedData.outcome` reference into a
filter-expression JSONPath and then fails to parse its own rewrite.
Test case YAML (per the documented shape):
```yaml
customEvaluations:
- label: "response contains estimat"
name: string_comparison
parameters:
- name: operator
value: contains
isReference: false
- name: actual
value: "$.generatedData.outcome"
isReference: true
- name: expected
value: "estimat"
isReference: false
```
### Actual result
The evaluation returns `status: ERROR` on every case:
```
"errorMessage": "Error parsing JSONPath
'$.outputs[?(@.type == 'general.echo')].payload.planner_response.lastExecution.outcome'"
```
i.e. the server maps `$.generatedData.outcome` to a path containing a
filter expression `[?(@.type == '...')]`, and the JSONPath implementation
evaluating it rejects filter syntax. The author's input is well-formed; the
failing path is generated internally.
### Expected result
Either the generated path parses, or `$.generatedData.outcome` maps to a
filter-free path, or the reference is rejected at test-create time with a
clear message. An ERROR status at run time on the documented reference makes
deterministic response-text assertions impossible in Testing Center.
### Why it matters
The LLM-judged `expectedOutcome` is lenient (see reproduction context in the
related issues: an agent whose instruction clause was removed still passed a
judge rubric that named the missing behavior). `string_comparison` is the
platform's only deterministic assertion over response text — precisely the
tool needed when the judge is too soft — and it errors as documented.
### Environment
- sf CLI:
- Org: Agentforce Developer Edition (`orgfarm-*.develop.my.salesforce.com`)
Contributor guide
Research direction
No source files or tests are named. Start by reproducing the custom string_comparison evaluation with the documented $.generatedData.outcome reference and inspect how the server rewrites and parses that path. Done means the reference evaluates successfully, maps to a parseable path, or is rejected at creation with a clear error.
Written by the indexing model from the issue text.
Assessment
- Domain
- cli, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100