huggingface / huggingface/paperswithcode-feedback

[Dataset Request] Add MetaTool (ToolE) — tool selection benchmark (ICLR 2024)

Open
#29 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
No language data
Stars
10
Forks
1
PR merge metrics
No merged PRs in 30d

Description

Hi! I'd like to request indexing the **MetaTool** benchmark and a leaderboard for the **Tool Selection** task.

### Paper
- **Title:** MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use
- **arXiv:** https://arxiv.org/abs/2310.03128
- **Venue:** ICLR 2024
- **Code / data:** https://github.com/HowieHwong/MetaTool

### Dataset
- **Source:** `github.com/HowieHwong/MetaTool` (ToolE dataset)
- **Size:** ~21K user queries mapped to ~43 tools
- **Sub-tasks:** (1) tool-usage awareness (should a tool be used), (2) tool selection — single-tool and multi-tool
- **Task:** Tool Selection
- **Primary metric:** **CSR — Correct Selection Rate (%)** = percentage of correct tool-selection actions

### Suggested results to seed the leaderboard
Verified from the arXiv HTML full text (https://arxiv.org/html/2310.03128). Nine LLMs evaluated; **GPT-4 was not evaluated on tool selection**. Task 1 = selection among *similar* tools; Task 3 = reliability (recognise when no tool applies). CSR (%):

| Model | Task 1 (0-shot) | Task 1 (5-shot) | Task 3 (0-shot) | Task 3 (5-shot) |
|---|---|---|---|---|
| ChatGPT (gpt-3.5-turbo) | 69.05 | 72.94 | 50.35 | 78.49 |
| ChatGLM2 | 54.17 | 57.44 | 6.63 | 15.68 |
| Llama2-7b | 45.95 | 51.12 | 0.90 | 2.51 |
| Llama2-13b | 44.06 | 49.85 | 2.31 | 5.93 |
| Vicuna-7b | 73.46 | 63.67 | 1.50 | 1.81 |
| Vicuna-13b | 58.23 | 63.15 | 2.51 | 3.42 |
| Vicuna-33b | 53.96 | 60.54 | 2.81 | 3.11 |
| Koala-13b | 56.34 | 60.85 | 1.70 | 5.83 |
| Baichuan2 | 59.12 | — | 32.26 | — |

The paper also reports a multi-tool selection table (Task 4) and scenario figures (Task 2, charts only — no extractable numbers). Source: arXiv 2310.03128. Can also be imported from `pwc-archive/evaluation-tables` if present.

### Note
Useful as a tool-selection counterpart to retrieval benchmarks like ToolRet/ToolBench. Thanks!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by inspecting the repository's existing dataset-indexing and leaderboard entry points, then review the ToolE data in github.com/HowieHwong/MetaTool and the arXiv HTML for the reported CSR results. Check pwc-archive/evaluation-tables if available. Done means MetaTool is indexed and its documented tool-selection results are represented in a leaderboard.

Written by the indexing model from the issue text.

Assessment

Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.