huggingface / huggingface/paperswithcode-feedback
[Dataset Request] Add ToolBench (API/tool retrieval) — ToolLLM (ICLR 2024)
- Dominant language
- No language data
- Stars
- 10
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
Hi! I'd like to request indexing the **ToolBench** tool-retrieval benchmark and a leaderboard for the **Tool Retrieval** task.
### Paper
- **Title:** ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs
- **arXiv:** https://arxiv.org/abs/2307.16789
- **Venue:** ICLR 2024 (Spotlight)
- **Code:** https://github.com/OpenBMB/ToolBench
### Dataset
- **HF (retrieval split):** [`NemoSheng/retrieval_toolbench`](https://huggingface.co/datasets/NemoSheng/retrieval_toolbench)
- **Size:** 451 test queries against a corpus of ~10,400 tool/API descriptions (RapidAPI ecosystem)
- **Settings:** I1 (single-tool instructions), I2 (intra-category multi-tool), I3 (intra-collection multi-tool)
- **Task:** Tool / API Retrieval
- **Primary metrics:** NDCG@1, NDCG@5 (retrieval); the paper's dense API Retriever (Sentence-BERT / BERT-base) is the reference system
### Suggested results to seed the leaderboard
Verified verbatim from **Table 2** of the paper (NDCG, %; cross-checked against the arXiv HTML and PDF). Note the paper's retrieval table uses **I1/I2/I3**, not the G1/G2/G3 labels used for the end-to-end solving leaderboard — do not conflate.
| Model | I1 @1 | I1 @5 | I2 @1 | I2 @5 | I3 @1 | I3 @5 |
|---|---|---|---|---|---|---|
| BM25 | 18.4 | 19.7 | 12.0 | 11.0 | 25.2 | 20.4 |
| OpenAI `text-embedding-ada-002` | 57.5 | 58.8 | 36.8 | 30.7 | 54.6 | 46.8 |
| **API Retriever (ToolLLM)** | **84.2** | **89.7** | **68.2** | **77.9** | **81.7** | **87.1** |
Source: arXiv PDF/HTML Table 2 — https://arxiv.org/pdf/2307.16789 . Additional entries can be imported from `pwc-archive/evaluation-tables` if present.
### Note
This retrieval split is widely reused as a tool-retrieval benchmark (also referenced by TR-bench, EMNLP 2024). Thanks!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the linked retrieval_toolbench dataset, the ToolBench repository, and Table 2 of the linked ToolLLM paper. Compare the repository's existing dataset and leaderboard conventions, then add the Tool Retrieval benchmark with I1/I2/I3 NDCG@1 and NDCG@5 results; done means the dataset and seeded results are indexed without conflating them with G1/G2/G3.
Written by the indexing model from the issue text.
Assessment
- Domain
- ai, data, search
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100