open-compass / open-compass/AgentCompass
[Bug] reconstruct_run_result drops StepInfo.tools from persisted trajectories
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 135
- Forks
- 28
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 37
Description
I noticed that reconstruct_run_result does not restore the tools field of persisted trajectory steps.
A serialized StepInfo can contain tool information such as:
{
"step_id": 1,
"tools": [
{
"name": "browser",
"description": "Browse a web page",
}
],
"assistant_content": {
"content": "Done",
"reasoning_content": "",
"tool_calls": [],
},
"metric": {},
"observation": [],
}
After passing this payload through reconstruct_run_result, the restored step contains:
restored.trajectory.steps[0].tools == []
The function restores assistant content, metrics, prompts, observations, and timestamps, but does not pass raw_step["tools"] to StepInfo.
This breaks lossless serialization and can make analysis results depend on whether they are produced during the original run or recomputed from persisted results. Analyzers that inspect step.tools, including the hack detection analyzer, may receive different inputs during an analysis-only rerun.
The expected behavior is for the reconstructed step to preserve the original tool information.
A possible fix is to restore the field when creating StepInfo:
tools=raw_step.get("tools", [])
A round-trip regression test could verify that serializing and reconstructing a trajectory preserves this field.
I reproduced this on revision 16cb375779271d4524cde785d1656e1fc4528fac with Python 3.11.9. No model or external API is required.
I'd be happy to work on the fix and add the regression test. Please let me know if this approach looks reasonable.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at reconstruct_run_result and the StepInfo construction it uses, then inspect the existing trajectory serialization path. Add coverage for a round trip containing StepInfo.tools and confirm the reconstructed step preserves the original tool information, including for analysis-only reruns.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100