deepset-ai / deepset-ai/haystack

LLMDocumentContentExtractor treats a chat generator's own "error" field as a failed call

Open
#12,734 0 comments 0 reactions 1 assignee View on GitHub

@julian-risch is already working on this.

Since Sep 13, 2026.

Dominant language
Python
Stars
26.6k
Forks
3.2k
Avg merge
1d 3h
Merged PRs (30d)
194

Description

LLMDocumentContentExtractor decides whether the per-document LLM call failed by checking if "error" in result on the chat generator's output dict. Generator outputs are free to carry extra fields, and a wrapper-style generator that always includes an error field (for example {"replies": [...], "error": None}) makes every document land in failed_documents with extraction_error=None, even though the call succeeded and returned usable content.

This is the same internal-sentinel collision class as the LLMMetadataExtractor bug fixed in #12685, on the other extractor.

Reproducer:

from unittest.mock import patch

from haystack import Document
from haystack.components.converters.image.document_to_image import DocumentToImageContent
from haystack.components.extractors.image import LLMDocumentContentExtractor
from haystack.dataclasses.chat_message import ChatMessage, ImageContent


class ErrorKeyChatGenerator:
    """Wrapper-style generator whose output dict always includes an "error" field."""

    def run(self, messages, **kwargs):
        return {"replies": [ChatMessage.from_assistant("extracted text")], "error": None}


extractor = LLMDocumentContentExtractor(chat_generator=ErrorKeyChatGenerator())

with patch.object(DocumentToImageContent, "run") as mock_convert:
    mock_convert.return_value = {
        "image_contents": [ImageContent.from_file_path("./test/test_files/images/apple.jpg")]
    }
    doc = Document(content="", meta={"file_path": "/path/to/image.pdf"})
    result = extractor.run(documents=[doc])

print({k: len(v) for k, v in result.items()})

Output (run twice, identical both times): {'documents': 0, 'failed_documents': 1} - the document is dropped with extraction_error=None even though the generator returned valid content. Expected: the document is processed and its content becomes "extracted text".

Proposed fix: in _process_llm_results, check "replies" not in result instead of "error" in result. The internal failure dicts never carry a replies key and generator outputs always do, so the failure sentinel no longer shares a namespace with generator output fields. PR follows.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.