salesforce / salesforce/evalon

Failure classification by name string misclassifies harness failures on name collision

Open
#11 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Scala
Stars
0
Forks
3
Avg merge
1d 21h
Merged PRs (30d)
2

Description

Summary

Failure classification in EvalonRunner.scala:87 decides "agent under test failed" vs "harness failed" by comparing the failed participant's name string to evaluatedName(scenario):

if participant == EvalonRunner.evaluatedName(scenario) then
  throw AgentStepFailedException(cause, t)   // exit code 1: agent failed
else throw SimulationFailedException(participant, cause, t)  // exit code 2: harness failed, retryable

If a harness participant shares the evaluated agent's name, its failure is misclassified.

Reachability

  • Participants can't collide with each other (they're Map keys), so the only exposure is an EventSourceConfig.name equal to the evaluated participant's name.
  • There is no scenario validation forbidding this (same gap as #9).
  • A failing event source sends ParticipantFailed(config.name, cause) (EventSourceActor.scala:115). If that name equals evaluatedName, EvalonRunner:87 throws AgentStepFailedException (exit code 1, "agent failed") instead of SimulationFailedException (exit code 2, "retryable harness failure").

Impact

Low, config-dependent, pre-existing. But the wrong exit code means automation/CI could blame the agent under test for an infrastructure outage, or skip a retry that a harness failure warrants.

Suggested fix

Classify by identity/role rather than by name string, e.g. carry a participant kind (evaluated vs simulated vs event-source) in ParticipantFailed so classification doesn't depend on names being distinct. Alternatively, validate at scenario load that event-source names and participant names are disjoint.

Found during review of #8. Related to #9 (validation gap) and the evaluated-participant-lookup dedup issue.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at EvalonRunner.scala:87 and trace ParticipantFailed from EventSourceActor.scala:115. Inspect how participant identity or role is represented and how scenario loading handles EventSourceConfig.name collisions. Done means a colliding event-source failure is classified as a retryable harness failure with exit code 2 rather than an agent failure with exit code 1.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.