[Feat]: A mechanism for submitting task feedback, or agent rewards
- Dominant language
- Shell
- Stars
- 25.7k
- Forks
- 2.6k
- Avg merge
- 3d 6h
- Merged PRs (30d)
- 16
Description
### Is your feature request related to a problem? Please describe.
Reinforcement learning is being used to train LLMs in agentic systems ([e.g.](https://verl.readthedocs.io/en/latest/sglang_multiturn/multiturn.html)), but the A2A spec does not expose any mechanism for clients to return rewards or feedback to the agent service. To enable continual improvement and self-learning for agents, the protocol should support:
1. Expose feedback as a capability, along with a schema soliciting feedback of a particular structure
2. An RPC route for submitting feedback to the server that conforms to the exposed schema
3. Specifying to which task response the feedback is to be associated.
4. Trust relationships for feedback, to avoid reward poisoning
The service can then use the submitted feedback as training data for model fine tuning, in cases where the underlying agent is backed by some model with learnable weights.
The third and fourth points above probably require some more explaining. Imagining the agent conversation as a Markov process, we might have a history as follows:
```
User Message -> Task 1: Agent response 1 -> User Message appending to Task 1 -> Task 1: Agent response 2
```
For the purposes of reinforcement learning rewards received after agent response (2) are worth less than those received after agent response (1) due to discounting. So, the task ID itself is insufficient for reward specification.
On the fourth point, agents that expose feedback as a capability may want to control entitlements for who is permitted to return feedback or rewards. One may imagine an attack on the agent whereby a network of coordinated users send malicious or incorrect feedback to an agent in an attempt to poison its training data.
### Describe the solution you'd like
## Expose `acceptsFeedback` as a capability
Agents should publish whether they accept feedback, and if they do, the specific form the expect that feedback to take. Here, one may modify the agent card like so:
```
export interface AgentCapabilities {
...,
/* Whether the agent accepts feedback. If non-null, a JSON schema specifying what form the feedback should take */
acceptsFeedback?: { [key: str]: any };
}
```
This would solve problem 1.
## Mechanism for returning feedback
One possible solution is to use `message/send` for returning feedback. The main issues I see here are:
1. Reward or feedback submission is semantically different than submitting messages, since the latter may create new tasks. Clients don't expect a response or new artifacts to be generated from the agent when submitting feedback, but they should expect new artifacts when submitting tasks or resubmitting or appending onto an existing task.
2. The agent service would need some way to differentiate between rewards (which don't require any new action) and client messages that create new tasks (which do require action).
Towards solving these two points, one approach may be to append `feedback: True` into the message metadata, or some other boolean flag indicating that the message doesn't require any action on the part of the agent, and that the agent can use to identify and log instances of feedback.
Another possible solution is to create dedicated RPC methods for feedback submission.
## Mechanism for specifying chronological ordering of rewards
Recalling again, the problem here is that a task might have multiple agent-user back and forth messages, and we need to be able to specify at which "turn" the reward was received.
Since tasks are stateful, one possibility here is to insist that feedback submission is always associated with the latest task response from the agent. This has the benefit of being immediately compatible with the current spec, but the drawback is that it precludes the possibility of submitting feedback at a particular point in the conversation after the conversation has completed.
Another possibility is to introduce unique identifiers on task responses, and to refer to these explicitly when submitting feedback.
## Authorization
This is addressed already by existing authentication schemes used by A2A.
### Describe alternatives you've considered
1. Implement a non-standard agent extension. This is possible, but I feel the direction of agentic training and systems makes the scope of this feature broadly applicable to the entire community. Agents should be able to improve themselves and there isn't a built-in mechanism to solicit feedback from clients right now.
2. Build separate APIs for processing rewards completely outside the A2A protocol.
### Additional context
_No response_
### Code of Conduct
- [x] I agree to follow this project's Code of Conduct
Contributor guide
Assessment
This issue has not been assessed yet.