salesforce / salesforce/evalon
RemoteAgent: evaluated agent's message sender can diverge from its participant name, corrupting the transcript and self-delivering messages
Nobody has claimed this yet.
- Dominant language
- Scala
- Stars
- 0
- Forks
- 3
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 2
Description
Summary
The evaluated agent's outgoing message carries a sender that the agent itself supplies. For RemoteAgent, that sender comes from the remote server's JSON response, but the remote is never told which participant name it is supposed to use. When the returned sender differs from the scenario's evaluated-participant name, the transcript is mislabeled, the transcript and the agent's own history diverge, and the runner delivers the agent's message back to the agent itself.
This is pre-existing on main and independent of #1 — none of the files involved are touched by that PR. Filing separately so it isn't conflated with the SimpleAgent work.
Root cause
The evaluated participant's name is known authoritatively by the framework: ScenarioRunner computes agentName and injects it into EvaluatedAgent (ScenarioRunner.scala:78-81, :96). But the sender that actually reaches the transcript is the one the agent stamps into Action.Send(Message(sender, …)), and each agent sources it differently:
ClaudeAgent— localagentNamector param (default"agent")RemoteAgent— whatever the remote server serializes intoMessage.sender(RemoteAgent.scala:71, deserialized fromStepResponse)
Critically, StepRequest sends protocol_version, history, events, respond_in — but not the agent's own participant name (RemoteAgentProtocol.scala:30-35). So a correct remote implementation has no reliable way to know what sender to return; it can only guess (e.g. scan history for its own prior turns).
Meanwhile the two readers of the sender disagree:
- History (
EvaluatedAgent.recordAction,:123) records withagentName. - Transcript + delivery + observer events (
ScenarioRunner.processSendAction,
:228,:235,:245) use the agent-suppliedmsg.sender.
Consequences when msg.sender != agentName
- Transcript mislabeled / diverges from history. The saved transcript attributes the message to the remote-supplied name, while the agent's own step history attributes it to
agentName. The judge reads the transcript, so it sees the wrong sender. - Self-delivery (correctness). The delivery loop skips a participant only when
name != msg.sender(ScenarioRunner.scala:235). With a mismatched sender, the evaluated agent (namedagentName) is no longer excluded and is in the conversation'sdirectConversations, so it receives its own outgoing message as an incoming turn on the next step — corrupting its history and potentially triggering an extra response cycle. - Observer events emit the wrong
sender(:245).
Reproduction (conceptual)
Run a scenario over endpoint: (→ RemoteAgent) whose evaluated participant is named e.g. assistant, with a remote server that returns Action.Send(Message(sender = "bot", content = "...")). Observe: transcript shows bot; agent history shows assistant; the agent receives its own message back.
Proposed fix
Make the framework the single source of truth for the evaluated agent's sender, since it already knows agentName. In EvaluatedAgent, stamp the sender before forwarding:
val stamped = action match
case Action.Send(message, toolTrace) => Action.Send(message.copy(sender = agentName), toolTrace)
case Action.End => Action.End
runner ! ScenarioRunner.ParticipantResponse(agentName, conversation, stamped)
This normalizes all three agent types uniformly (and makes ClaudeAgent's agentName param and any agent-supplied sender vestigial). Alternatively, have processSendAction use the participant value it's already passed (which the End branch already does) instead of msg.sender.
Notes / related
- #1 adds a third sender source (the
SimpleAgentadapter, viaevaluatedName(scenario)) and duplicates theevaluatedNamelookup already in ScenarioRunner:78-81. Centralizing per this fix would let that adapter drop itsagentNameparameter entirely. - Scope note: the fix touches
EvaluatedAgent(and optionallyClaudeAgent), so it's intentionally not bundled into #1.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in EvaluatedAgent where actions are forwarded, then trace ScenarioRunner.processSendAction and the sender handling at lines 228, 235, and 245. Review ScenarioRunner.scala:78-81 and :96, plus RemoteAgentProtocol.scala:30-35 and RemoteAgent.scala:71, to confirm the authoritative participant name. Done means transcript, history, delivery, and observer events consistently use the evaluated participant name without self-delivery.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 74/100