huggingface / huggingface/paperswithcode-feedback
[Dataset Request] Add ToolRet — tool retrieval benchmark (ACL 2025)
- Dominant language
- No language data
- Stars
- 10
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
Hi! I'd like to request indexing the **ToolRet** benchmark and creating a leaderboard for the **Tool Retrieval** task.
### Paper
- **Title:** Retrieval Models Aren't Tool-Savvy: Benchmarking Tool Retrieval for Large Language Models
- **arXiv:** https://arxiv.org/abs/2503.01763
- **Venue:** ACL 2025 (Findings)
- **Code:** https://github.com/mangopy/benchmarking-tool-retrieval
- **Project page:** https://mangopy.github.io/tool-retrieval-benchmark/
### Dataset
- **HF datasets:** [`mangopy/ToolRet-Queries`](https://huggingface.co/datasets/mangopy/ToolRet-Queries), [`mangopy/ToolRet-Tools`](https://huggingface.co/datasets/mangopy/ToolRet-Tools)
- **Size:** ~7,600 retrieval tasks across 35 sub-tasks; corpus of ~43K tools
- **Subsets:** ToolRet-web (Web APIs), ToolRet-code (code functions), ToolRet-customized (NL-described apps)
- **Task:** Tool Retrieval (information retrieval over a tool corpus)
- **Primary metrics:** nDCG@10, Completeness/Recall@K
### Suggested results to seed the leaderboard
The paper benchmarks six families of IR models (BM25, dense retrievers, instruction-tuned embedders, etc.). Headline finding, directly from the paper:
| Model | nDCG@10 |
|---|---|
| NV-Embed-v1 (best in paper) | 33.83 |
Full per-model / per-subset numbers are in the results table of the paper — happy to point at the exact table, and they can also be imported from the legacy `pwc-archive/evaluation-tables` if present.
### Note
The key takeaway is that models strong on conventional IR benchmarks perform poorly here, so this is a valuable standalone leaderboard. Thanks!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by inspecting the repository structure and any existing dataset or leaderboard request conventions; no implementation files or tests are named in the issue. Verify the linked ToolRet paper, Hugging Face datasets, and result tables, then determine the project's required indexing and leaderboard format. Done means ToolRet is indexed and its retrieval metrics and seed results are represented in the supported leaderboard.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- huggingface
- Domain
- data, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100