allenai / allenai/asta-plugins
semantic-scholar: paper claim-verification routes to general web search instead of asta papers snippet-search
- Lenguaje dominante
- Python
- Estrellas
- 39
- Forks
- 6
- Merge medio
- 9 h 33 min
- PR fusionados (30 d)
- 8
Descripción
## Behavior gap: paper/literature claim-verification routes to general web search instead of Asta paper tools
**Skill(s):** `asta-tools/semantic-scholar` (primary), `asta-tools/find-literature` (secondary)
### Prompt / context
An agent was verifying a specific factual claim about a known paper — whether the
deepfigures figure-extraction paper (*Extracting Scientific Figures with Distantly
Supervised Neural Networks*, Siegel et al., JCDL 2018 — arXiv:1804.02445,
S2 `1fec9d41d372267b4474f18cbeadd806c8b67adb`) ablates **caption-input conditioning**.
This is exactly the shape the `semantic-scholar` skill targets: a named paper + a
specific claim to confirm against its full text.
### Current behavior
The agent reached for **general web-search tools** to answer the query. It only used
Asta's Semantic Scholar tooling (`asta papers get` + `asta papers snippet-search`)
after the operator explicitly said *"try using asta instead of the websearch tools."*
On the second pass the Asta path worked well — `snippet-search` pulled the paper's
"Matching Captions" section verbatim and settled the question directly against the
paper text.
### Desired behavior
For a query that names a specific paper (or DOI/arXiv/S2 ID) and asks to confirm a
claim, method, or result within it, the agent should **prefer `asta papers` first**
— `get` for metadata, `snippet-search` for in-body claim verification — rather than
general web search, without needing the operator to redirect it.
### Hypothesized root cause
The `semantic-scholar` SKILL.md already lists this case ("find specific claims,
methods, or evidence within paper full text — use `snippet-search`"), so the gap is
**routing under competition with a zero-friction built-in `web_search` tool**, not a
missing description. Two contributing factors to test:
1. The skill description has no explicit *"prefer over general web search"* cue, so
for claim-verification framing the always-available `web_search` wins by default.
2. The claim-verification / fact-check use-case is phrased as "find specific claims"
— it may not trigger on prompts framed as *verify / confirm / is it true that…*
about a named paper.
### Proposed fix (to validate via the eval loop)
Strengthen `semantic-scholar` skill routing: add an explicit
"prefer over general web search for questions about a specific paper's contents /
claims / results" trigger, and surface `snippet-search` as the first tool for
verifying or fact-checking a statement against a named paper's full text.
### Status / next step
This is a **step-1 report** capture. The eval-backed TDD loop (baseline on
`origin/main` → skill edit → PR arm → regression guards, per the improve-skills
WORKFLOW) requires Docker + `inspect` evals against `asta-bench-private` /
`agent-baselines` and cannot run inside a gas2own dispatcher turn — it is the
follow-up (async job) once this gap is confirmed worth fixing. A new
`asta_skills` case would assert the agent calls `asta papers snippet-search`
(not `web_search`) when verifying a claim about a named paper.
Suggested-by: @jbragg
Guía de contribución
No hay ninguna guía de contribución indexada para este repositorio
Evaluación
Este issue todavía no se ha evaluado.