Data conamination in build repair agent evaluation
Open
Nobody has claimed this yet.
CI agents
evals
hackbot
- Dominant language
- Python
- Stars
- 570
- Forks
- 351
- Avg merge
- 2d 13h
- Merged PRs (30d)
- 65
Description
We used to sanitize Bugzilla data manually to run evals for the build repair agent. Now it's using the MCP so that it can see the fixes in the historical dataset. There should be a way to limit Bugzilla comments to the ones before the fix.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the MCP Bugzilla data path used by the build repair agent evaluation. Determine how the fix point is represented and where historical comments are selected, then verify that comments after the fix are excluded while earlier comments remain available.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100