zapier / zapier/AutomationBench
EvalScope integration: one-command evaluation support for AutomationBench
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 290
- Forks
- 45
- PR merge metrics
- No merged PRs in 30d
Description
Hi zapier/AutomationBench maintainers,
EvalScope now supports AutomationBench. This gives users another way to run the benchmark when they want a standardized evaluation workflow around OpenAI-compatible model endpoints, saved predictions, scoring, and report generation.
This may be useful for users who find this repository while setting up AutomationBench and want a quick path to evaluate a hosted or locally served model.
Example:
evalscope eval \
--model YOUR_MODEL \
--api-url OPENAI_API_COMPAT_URL \
--api-key EMPTY_TOKEN \
--datasets automation_bench \
--limit 10
Docs: https://evalscope.readthedocs.io/en/latest/benchmarks/automation_bench.html
EvalScope repo: https://github.com/modelscope/evalscope
We can also open a small PR adding this as a README/Docs note, linked to this issue.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Review the repository README and the linked EvalScope AutomationBench documentation first. Add a concise README or documentation note covering the EvalScope command and links provided in the issue; done means users can find and follow the documented evaluation path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 1/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 70/100