huggingface / huggingface/paperswithcode-feedback
[Dataset Request] Add MetaTool (ToolE) — tool selection benchmark (ICLR 2024)
- Dominant language
- No language data
- Stars
- 10
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
Hi! I'd like to request indexing the **MetaTool** benchmark and a leaderboard for the **Tool Selection** task.
### Paper
- **Title:** MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use
- **arXiv:** https://arxiv.org/abs/2310.03128
- **Venue:** ICLR 2024
- **Code / data:** https://github.com/HowieHwong/MetaTool
### Dataset
- **Source:** `github.com/HowieHwong/MetaTool` (ToolE dataset)
- **Size:** ~21K user queries mapped to ~43 tools
- **Sub-tasks:** (1) tool-usage awareness (should a tool be used), (2) tool selection — single-tool and multi-tool
- **Task:** Tool Selection
- **Primary metric:** **CSR — Correct Selection Rate (%)** = percentage of correct tool-selection actions
### Suggested results to seed the leaderboard
Verified from the arXiv HTML full text (https://arxiv.org/html/2310.03128). Nine LLMs evaluated; **GPT-4 was not evaluated on tool selection**. Task 1 = selection among *similar* tools; Task 3 = reliability (recognise when no tool applies). CSR (%):
| Model | Task 1 (0-shot) | Task 1 (5-shot) | Task 3 (0-shot) | Task 3 (5-shot) |
|---|---|---|---|---|
| ChatGPT (gpt-3.5-turbo) | 69.05 | 72.94 | 50.35 | 78.49 |
| ChatGLM2 | 54.17 | 57.44 | 6.63 | 15.68 |
| Llama2-7b | 45.95 | 51.12 | 0.90 | 2.51 |
| Llama2-13b | 44.06 | 49.85 | 2.31 | 5.93 |
| Vicuna-7b | 73.46 | 63.67 | 1.50 | 1.81 |
| Vicuna-13b | 58.23 | 63.15 | 2.51 | 3.42 |
| Vicuna-33b | 53.96 | 60.54 | 2.81 | 3.11 |
| Koala-13b | 56.34 | 60.85 | 1.70 | 5.83 |
| Baichuan2 | 59.12 | — | 32.26 | — |
The paper also reports a multi-tool selection table (Task 4) and scenario figures (Task 2, charts only — no extractable numbers). Source: arXiv 2310.03128. Can also be imported from `pwc-archive/evaluation-tables` if present.
### Note
Useful as a tool-selection counterpart to retrieval benchmarks like ToolRet/ToolBench. Thanks!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by inspecting the repository's existing dataset-indexing and leaderboard entry points, then review the ToolE data in github.com/HowieHwong/MetaTool and the arXiv HTML for the reported CSR results. Check pwc-archive/evaluation-tables if available. Done means MetaTool is indexed and its documented tool-selection results are represented in a leaderboard.
Written by the indexing model from the issue text.
Assessment
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100