neulab / neulab/agent-data-protocol
Normalize observation-only steps without blank agent messages
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 193
- Forks
- 17
- PR merge metrics
- No merged PRs in 30d
Description
Some standardized samples represent environment/tool observations as blank source: agent steps with only an observation field. This is schema-valid, but semantically awkward because the environment, not the agent, authored the observation.
Examples from the audit/heuristics include web/mobile datasets such as WebLINX, Wonderbread, Mind2Web, Go-Browse-WA, TurkingBench, WebArena, AndroidControl, and related browser/mobile traces.
Acceptance criteria:
- Decide the canonical representation for standalone observations when no preceding tool call is available.
- Prefer attaching observations to the triggering tool-call step when possible.
- If standalone observation steps remain necessary, document the convention clearly or add schema support for environment-authored steps.
- Regenerate affected samples after the representation is settled.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the audit/heuristics examples and the affected WebLINX, Wonderbread, Mind2Web, Go-Browse-WA, TurkingBench, WebArena, and AndroidControl samples. Compare standalone observations with observations attached to tool calls, then settle and document one representation or add environment-authored schema support. Done means the convention is clear and affected samples are regenerated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100