salesforce / salesforce/evalon

RemoteAgent: evaluated agent's message sender can diverge from its participant name, corrupting the transcript and self-delivering messages

Open
#2 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Scala
Stars
0
Forks
3
Avg merge
1d 21h
Merged PRs (30d)
2

Description

Summary

The evaluated agent's outgoing message carries a sender that the agent itself supplies. For RemoteAgent, that sender comes from the remote server's JSON response, but the remote is never told which participant name it is supposed to use. When the returned sender differs from the scenario's evaluated-participant name, the transcript is mislabeled, the transcript and the agent's own history diverge, and the runner delivers the agent's message back to the agent itself.

This is pre-existing on main and independent of #1 — none of the files involved are touched by that PR. Filing separately so it isn't conflated with the SimpleAgent work.

Root cause

The evaluated participant's name is known authoritatively by the framework: ScenarioRunner computes agentName and injects it into EvaluatedAgent (ScenarioRunner.scala:78-81, :96). But the sender that actually reaches the transcript is the one the agent stamps into Action.Send(Message(sender, …)), and each agent sources it differently:

  • ClaudeAgent — local agentName ctor param (default "agent")
  • RemoteAgent — whatever the remote server serializes into Message.sender (RemoteAgent.scala:71, deserialized from StepResponse)

Critically, StepRequest sends protocol_version, history, events, respond_in — but not the agent's own participant name (RemoteAgentProtocol.scala:30-35). So a correct remote implementation has no reliable way to know what sender to return; it can only guess (e.g. scan history for its own prior turns).

Meanwhile the two readers of the sender disagree:

  • History (EvaluatedAgent.recordAction, :123) records with agentName.
  • Transcript + delivery + observer events (ScenarioRunner.processSendAction,
    :228, :235, :245) use the agent-supplied msg.sender.

Consequences when msg.sender != agentName

  1. Transcript mislabeled / diverges from history. The saved transcript attributes the message to the remote-supplied name, while the agent's own step history attributes it to agentName. The judge reads the transcript, so it sees the wrong sender.
  2. Self-delivery (correctness). The delivery loop skips a participant only when name != msg.sender (ScenarioRunner.scala:235). With a mismatched sender, the evaluated agent (named agentName) is no longer excluded and is in the conversation's directConversations, so it receives its own outgoing message as an incoming turn on the next step — corrupting its history and potentially triggering an extra response cycle.
  3. Observer events emit the wrong sender (:245).

Reproduction (conceptual)

Run a scenario over endpoint: (→ RemoteAgent) whose evaluated participant is named e.g. assistant, with a remote server that returns Action.Send(Message(sender = "bot", content = "...")). Observe: transcript shows bot; agent history shows assistant; the agent receives its own message back.

Proposed fix

Make the framework the single source of truth for the evaluated agent's sender, since it already knows agentName. In EvaluatedAgent, stamp the sender before forwarding:

val stamped = action match
  case Action.Send(message, toolTrace) => Action.Send(message.copy(sender = agentName), toolTrace)
  case Action.End                       => Action.End
runner ! ScenarioRunner.ParticipantResponse(agentName, conversation, stamped)

This normalizes all three agent types uniformly (and makes ClaudeAgent's agentName param and any agent-supplied sender vestigial). Alternatively, have processSendAction use the participant value it's already passed (which the End branch already does) instead of msg.sender.

Notes / related

  • #1 adds a third sender source (the SimpleAgent adapter, via evaluatedName(scenario)) and duplicates the evaluatedName lookup already in ScenarioRunner:78-81. Centralizing per this fix would let that adapter drop its agentName parameter entirely.
  • Scope note: the fix touches EvaluatedAgent (and optionally ClaudeAgent), so it's intentionally not bundled into #1.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in EvaluatedAgent where actions are forwarded, then trace ScenarioRunner.processSendAction and the sender handling at lines 228, 235, and 245. Review ScenarioRunner.scala:78-81 and :96, plus RemoteAgentProtocol.scala:30-35 and RemoteAgent.scala:71, to confirm the authoritative participant name. Done means transcript, history, delivery, and observer events consistently use the evaluated participant name without self-delivery.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.