posit-dev / posit-dev/chatlas

Anthropic document_index citations don't account for documents embedded in custom tool results

Open
#381 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

ai-triage:done Priority: Low
Dominant language
Python
Stars
176
Forks
28
Avg merge
18h 42m
Merged PRs (30d)
16

Description

Context

Follow-up from #375-adjacent work in fix/anthropic-web-fetch-citations (anthropic_document_sources() in chatlas/_provider_anthropic.py), which resolves Anthropic document_index citations to source URLs across the whole request.

Gap

anthropic_document_sources() walks each turn's contents and assigns a document_index slot for ContentToolResponseFetch, ContentPDF, ContentDocument, and non-image ContentUploaded. It does not look inside ContentToolResult.get_model_value().

A custom tool can return ContentToolResult(value=..., model_format="as_is") whose value is a raw Anthropic block list containing a DocumentBlockParam or SearchResultBlockParam. That block is sent to Anthropic as part of the tool result content and occupies a document_index slot server-side, but anthropic_document_sources() never accounts for it — so every document_index computed for citations after that tool result is off, and citations pointing at the tool-embedded document can never resolve to a source.

Reproduction sketch

  1. Register a custom tool that returns ContentToolResult(value=[{"type": "document", ...}], model_format="as_is").
  2. Follow it with a web_fetch or another document-producing turn.
  3. Citations for content after the tool result resolve to the wrong WebSource (or None) because the tool-embedded document's slot wasn't counted.

Notes

This is a narrow/advanced case — it requires a custom tool to hand-construct raw Anthropic document/search-result blocks (not part of chatlas's documented tool-result patterns) — so it was shipped as a known limitation rather than blocking the citation-resolution fix. Fixing it would mean extending anthropic_document_sources() to also scan ContentToolResult's raw content for document-shaped blocks.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in chatlas/_provider_anthropic.py at anthropic_document_sources() and inspect how ContentToolResult.get_model_value() is represented in the request contents. Reproduce the custom as_is tool result followed by a document-producing turn, then verify that embedded document and search-result blocks consume document_index slots and later citations resolve to the correct WebSource.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.