salesforce / salesforce/evalon
Failure classification by name string misclassifies harness failures on name collision
Nobody has claimed this yet.
- Dominant language
- Scala
- Stars
- 0
- Forks
- 3
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 2
Description
Summary
Failure classification in EvalonRunner.scala:87 decides "agent under test failed" vs "harness failed" by comparing the failed participant's name string to evaluatedName(scenario):
if participant == EvalonRunner.evaluatedName(scenario) then
throw AgentStepFailedException(cause, t) // exit code 1: agent failed
else throw SimulationFailedException(participant, cause, t) // exit code 2: harness failed, retryable
If a harness participant shares the evaluated agent's name, its failure is misclassified.
Reachability
- Participants can't collide with each other (they're
Mapkeys), so the only exposure is anEventSourceConfig.nameequal to the evaluated participant's name. - There is no scenario validation forbidding this (same gap as #9).
- A failing event source sends
ParticipantFailed(config.name, cause)(EventSourceActor.scala:115). If that name equalsevaluatedName,EvalonRunner:87throwsAgentStepFailedException(exit code 1, "agent failed") instead ofSimulationFailedException(exit code 2, "retryable harness failure").
Impact
Low, config-dependent, pre-existing. But the wrong exit code means automation/CI could blame the agent under test for an infrastructure outage, or skip a retry that a harness failure warrants.
Suggested fix
Classify by identity/role rather than by name string, e.g. carry a participant kind (evaluated vs simulated vs event-source) in ParticipantFailed so classification doesn't depend on names being distinct. Alternatively, validate at scenario load that event-source names and participant names are disjoint.
Found during review of #8. Related to #9 (validation gap) and the evaluated-participant-lookup dedup issue.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at EvalonRunner.scala:87 and trace ParticipantFailed from EventSourceActor.scala:115. Inspect how participant identity or role is represented and how scenario loading handles EventSourceConfig.name collisions. Done means a colliding event-source failure is classified as a retryable harness failure with exit code 2 rather than an agent failure with exit code 1.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 50/100