huggingface / huggingface/paperswithcode-feedback

[Dataset Request] Add ToolBench (API/tool retrieval) — ToolLLM (ICLR 2024)

Open
#28 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
No language data
Stars
10
Forks
1
PR merge metrics
No merged PRs in 30d

Description

Hi! I'd like to request indexing the **ToolBench** tool-retrieval benchmark and a leaderboard for the **Tool Retrieval** task.

### Paper
- **Title:** ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs
- **arXiv:** https://arxiv.org/abs/2307.16789
- **Venue:** ICLR 2024 (Spotlight)
- **Code:** https://github.com/OpenBMB/ToolBench

### Dataset
- **HF (retrieval split):** [`NemoSheng/retrieval_toolbench`](https://huggingface.co/datasets/NemoSheng/retrieval_toolbench)
- **Size:** 451 test queries against a corpus of ~10,400 tool/API descriptions (RapidAPI ecosystem)
- **Settings:** I1 (single-tool instructions), I2 (intra-category multi-tool), I3 (intra-collection multi-tool)
- **Task:** Tool / API Retrieval
- **Primary metrics:** NDCG@1, NDCG@5 (retrieval); the paper's dense API Retriever (Sentence-BERT / BERT-base) is the reference system

### Suggested results to seed the leaderboard
Verified verbatim from **Table 2** of the paper (NDCG, %; cross-checked against the arXiv HTML and PDF). Note the paper's retrieval table uses **I1/I2/I3**, not the G1/G2/G3 labels used for the end-to-end solving leaderboard — do not conflate.

| Model | I1 @1 | I1 @5 | I2 @1 | I2 @5 | I3 @1 | I3 @5 |
|---|---|---|---|---|---|---|
| BM25 | 18.4 | 19.7 | 12.0 | 11.0 | 25.2 | 20.4 |
| OpenAI `text-embedding-ada-002` | 57.5 | 58.8 | 36.8 | 30.7 | 54.6 | 46.8 |
| **API Retriever (ToolLLM)** | **84.2** | **89.7** | **68.2** | **77.9** | **81.7** | **87.1** |

Source: arXiv PDF/HTML Table 2 — https://arxiv.org/pdf/2307.16789 . Additional entries can be imported from `pwc-archive/evaluation-tables` if present.

### Note
This retrieval split is widely reused as a tool-retrieval benchmark (also referenced by TR-bench, EMNLP 2024). Thanks!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the linked retrieval_toolbench dataset, the ToolBench repository, and Table 2 of the linked ToolLLM paper. Compare the repository's existing dataset and leaderboard conventions, then add the Tool Retrieval benchmark with I1/I2/I3 NDCG@1 and NDCG@5 results; done means the dataset and seeded results are indexed without conflating them with G1/G2/G3.

Written by the indexing model from the issue text.

Assessment

Domain
ai, data, search
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.