salesforce / salesforce/evalon
Agent step failures should not be swallowed as empty Action.Send
Nobody has claimed this yet.
- Dominant language
- Scala
- Stars
- 0
- Forks
- 3
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 2
Description
Summary
When Agent.step fails, EvaluatedAgent logs the error and pretends the agent sent an empty message:
ctx.pipeToSelf(agent.step(history, events, respondIn)) {
case Success(action) => AgentResult(action, respondIn)
case Failure(e) =>
ctx.log.error("Agent step failed", e)
AgentResult(Action.send(agentName, ""), respondIn)
}
That is not a real reply. Empty send is skipped later, so a crash looks like silence. The simulation keeps going (or hangs until timeout) instead of failing fast.
This also undoes SimpleAgent.toAgent, which now returns Future.failed for a null CompletionStage / thrown step. That failure is converted back into an empty send here.
Current behavior
- Nothing in the transcript.
ScenarioRunner.processSendActiondrops empty content (no message, no delivery, no turn count). The judge never sees that the agent failed. - Agent history ignores it too. Record Action skips empty sends, so the next step has no record of the failure.
- The run does not fail. EvalonRunner is still waiting on SimulationResult. Other participants keep talking. Typical outcomes:
- simulated user eventually [END]s → eval scores a silent/broken agent
- nobody ends → wait until simulation timeout
- Asymmetric with [END]. Explicit end is recorded as Signals.End. A crash is quieter than a deliberate end.
Proposed options
A. Fail the simulation
On Failure, send a distinct result to ScenarioRunner (e.g. AgentFailed(cause) or Action plus error). Runner stops, EvalonRunner.run fails with that cause (or EvalonResult carries the error). Fast, honest, matches library expectations.
B. Treat failure as Action.End
Same stop as [END], plus transcript note ([AGENT_ERROR] / error text). Run completes; judge can score an aborted conversation. Weaker than A (looks like a normal end unless the marker is obvious).
C. Retry then A or B
One retry on 5xx/timeout-style failures, then fail or end. More moving parts; only worth it if remote agents are flaky.
D. Make policy configurable
EvalonRunOptions.withOnAgentFailure(Fail | End | EmptySend) with Fail as default. Keeps today’s swallow only as an explicit opt-in.
Test plan (once fixed)
- SimpleAgent.step returns null → failed future → simulation fails (or recorded end), not a hang / empty skip
- step throws or CompletionStage completes exceptionally → same
- Successful Action.End and non-empty Send unchanged
- Empty Send from a successful step can stay skipped (that is a real “say nothing”)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at EvaluatedAgent's Agent.step failure handler, then trace ScenarioRunner.processSendAction and EvalonRunner.run to understand how empty sends, end signals, and simulation failures propagate. Review SimpleAgent.toAgent and the listed failure cases before choosing and documenting one failure policy. Done means thrown or exceptionally completed steps no longer become silent empty sends, while successful End, non-empty Send, and intentional empty Send retain their stated behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100