microsoft / microsoft/STATE-Bench
Agent Learning Track compliance: deterministic use of retrieved learnings before domain-tool execution
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 91
- Forks
- 13
- PR merge metrics
- No merged PRs in 30d
Description
We are planning a paired Agent Learning Track study using a custom BaseLLMClient + BaseAgent.
Both conditions would:
- build the learning artifact only from
datasets/train_task_trajectories/; - expose the same read-only
retrieve_learnings(query, top_k=3) -> list[str]; - present the same retrieved strings to the agent model;
- leave benchmark prompts, protocol files, task files, environments, domain tools, tool names, and argument keys unchanged; and
- let the STATE-Bench harness execute every domain tool.
The treatment condition adds deterministic middleware inside the custom BaseAgent, after the model proposes a domain-tool call but before the agent returns its AgentToolCallRequest. The middleware reads only:
- learning strings already returned by
retrieve_learnings, and - tool results already present in the current canonical conversation.
When a retrieved relation names a protected argument and a unique, type-compatible witness exists in the current run, the middleware may replace that argument value before returning the request. If the relation does not apply, the request passes through unchanged. It does not call the provided domain tool_handlers, inspect held-out task definitions or requirements, or mutate benchmark state. The benchmark still validates and executes the resulting request normally.
Issue #26 indicates that tools related to retrieving and using learnings may be added or modified. Would this deterministic pre-execution use of retrieved learnings be compliant with the Agent Learning Track and eligible for the official leaderboard, or must learnings affect domain-tool calls only through the model interpretation of the returned strings?
If compliant, is any additional method or trajectory-metadata disclosure required for submission?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the custom BaseLLMClient and BaseAgent references, datasets/train_task_trajectories/, issue #26, and the STATE-Bench harness behavior described here. The work is complete when maintainers clarify whether deterministic middleware may alter domain-tool calls and state any required submission or trajectory-metadata disclosure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, testing
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100