allenai / allenai/asta-plugins

semantic-scholar: paper claim-verification routes to general web search instead of asta papers snippet-search

Abierto
#125 5 comentarios 0 reacciones 0 asignados Ver en GitHub
enhancement
Lenguaje dominante
Python
Estrellas
39
Forks
6
Merge medio
9 h 33 min
PR fusionados (30 d)
8

Descripción

## Behavior gap: paper/literature claim-verification routes to general web search instead of Asta paper tools

**Skill(s):** `asta-tools/semantic-scholar` (primary), `asta-tools/find-literature` (secondary)

### Prompt / context

An agent was verifying a specific factual claim about a known paper — whether the
deepfigures figure-extraction paper (*Extracting Scientific Figures with Distantly
Supervised Neural Networks*, Siegel et al., JCDL 2018 — arXiv:1804.02445,
S2 `1fec9d41d372267b4474f18cbeadd806c8b67adb`) ablates **caption-input conditioning**.
This is exactly the shape the `semantic-scholar` skill targets: a named paper + a
specific claim to confirm against its full text.

### Current behavior

The agent reached for **general web-search tools** to answer the query. It only used
Asta's Semantic Scholar tooling (`asta papers get` + `asta papers snippet-search`)
after the operator explicitly said *"try using asta instead of the websearch tools."*
On the second pass the Asta path worked well — `snippet-search` pulled the paper's
"Matching Captions" section verbatim and settled the question directly against the
paper text.

### Desired behavior

For a query that names a specific paper (or DOI/arXiv/S2 ID) and asks to confirm a
claim, method, or result within it, the agent should **prefer `asta papers` first**
— `get` for metadata, `snippet-search` for in-body claim verification — rather than
general web search, without needing the operator to redirect it.

### Hypothesized root cause

The `semantic-scholar` SKILL.md already lists this case ("find specific claims,
methods, or evidence within paper full text — use `snippet-search`"), so the gap is
**routing under competition with a zero-friction built-in `web_search` tool**, not a
missing description. Two contributing factors to test:

1. The skill description has no explicit *"prefer over general web search"* cue, so
for claim-verification framing the always-available `web_search` wins by default.
2. The claim-verification / fact-check use-case is phrased as "find specific claims"
— it may not trigger on prompts framed as *verify / confirm / is it true that…*
about a named paper.

### Proposed fix (to validate via the eval loop)

Strengthen `semantic-scholar` skill routing: add an explicit
"prefer over general web search for questions about a specific paper's contents /
claims / results" trigger, and surface `snippet-search` as the first tool for
verifying or fact-checking a statement against a named paper's full text.

### Status / next step

This is a **step-1 report** capture. The eval-backed TDD loop (baseline on
`origin/main` → skill edit → PR arm → regression guards, per the improve-skills
WORKFLOW) requires Docker + `inspect` evals against `asta-bench-private` /
`agent-baselines` and cannot run inside a gas2own dispatcher turn — it is the
follow-up (async job) once this gap is confirmed worth fixing. A new
`asta_skills` case would assert the agent calls `asta papers snippet-search`
(not `web_search`) when verifying a claim about a named paper.

Suggested-by: @jbragg

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.