AMD-AGI / AMD-AGI/AgentKernelArena
Add Holdout Test Evaluation via Independent Agent
Open
@irvineoy is already working on this.
Since Jan 26, 2026.
- Dominant language
- Python
- Stars
- 118
- Forks
- 14
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 8
Description
Holdout tests are more rigorous evaluations and are intentionally hidden from the agent being tested.
We should add support in the framework to run these holdout tests. For each optimized task implementation, the framework would launch an independent third-party agent to evaluate the optimized code using the holdout tests and generate a final evaluation report. This helps ensure a more robust and unbiased assessment of both correctness and performance.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.