huggingface / huggingface/paperswithcode-feedback

[Dataset Request] Add ToolRet — tool retrieval benchmark (ACL 2025)

Open
#27 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
No language data
Stars
10
Forks
1
PR merge metrics
No merged PRs in 30d

Description

Hi! I'd like to request indexing the **ToolRet** benchmark and creating a leaderboard for the **Tool Retrieval** task.

### Paper
- **Title:** Retrieval Models Aren't Tool-Savvy: Benchmarking Tool Retrieval for Large Language Models
- **arXiv:** https://arxiv.org/abs/2503.01763
- **Venue:** ACL 2025 (Findings)
- **Code:** https://github.com/mangopy/benchmarking-tool-retrieval
- **Project page:** https://mangopy.github.io/tool-retrieval-benchmark/

### Dataset
- **HF datasets:** [`mangopy/ToolRet-Queries`](https://huggingface.co/datasets/mangopy/ToolRet-Queries), [`mangopy/ToolRet-Tools`](https://huggingface.co/datasets/mangopy/ToolRet-Tools)
- **Size:** ~7,600 retrieval tasks across 35 sub-tasks; corpus of ~43K tools
- **Subsets:** ToolRet-web (Web APIs), ToolRet-code (code functions), ToolRet-customized (NL-described apps)
- **Task:** Tool Retrieval (information retrieval over a tool corpus)
- **Primary metrics:** nDCG@10, Completeness/Recall@K

### Suggested results to seed the leaderboard
The paper benchmarks six families of IR models (BM25, dense retrievers, instruction-tuned embedders, etc.). Headline finding, directly from the paper:

| Model | nDCG@10 |
|---|---|
| NV-Embed-v1 (best in paper) | 33.83 |

Full per-model / per-subset numbers are in the results table of the paper — happy to point at the exact table, and they can also be imported from the legacy `pwc-archive/evaluation-tables` if present.

### Note
The key takeaway is that models strong on conventional IR benchmarks perform poorly here, so this is a valuable standalone leaderboard. Thanks!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by inspecting the repository structure and any existing dataset or leaderboard request conventions; no implementation files or tests are named in the issue. Verify the linked ToolRet paper, Hugging Face datasets, and result tables, then determine the project's required indexing and leaderboard format. Done means ToolRet is indexed and its retrieval metrics and seed results are represented in the supported leaderboard.

Written by the indexing model from the issue text.

Assessment

Tech stack
huggingface
Domain
data, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.