docling-project / docling-project/docling-agent

RAG: _attempt_answer crashes on unparseable model output

Open Beginner friendly
#52 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
166
Forks
21
PR merge metrics
No merged PRs in 30d

Description

`_attempt_answer` in `docling_agent/agent/rag.py` parses the model response without any check:

```python
d = find_json_dicts(answer)[0]
return AnswerAttempt(can_answer=d["can_answer"], response=d["response"])
```

`_select_section`, in the same file, guards the exact same pattern and falls back to
picking the first unvisited section. The other `find_json_dicts` call sites
(orchestrator, enricher, editor) all check the result too. This is the only
unguarded one.

It can trigger on every backend. On the mellea backend the retry budget is best
effort: when rejection sampling runs out, the session returns a failed sample
instead of raising. On the direct HTTP backends (Ollama and the OpenAI-compatible
ones), requirements are not enforced at all, so the first response without a valid
JSON block reaches the parse directly.
Either way this line raises `IndexError` (no JSON found) or `KeyError` (missing key)
and the whole RAG run aborts. Without the crash, `_rag_loop` would have moved on to
the next section, or returned the partial answer it already had.

Fix, mirroring `_select_section`:

```python
dicts = find_json_dicts(answer)
d = dicts[0] if dicts else {}
if not isinstance(d.get("can_answer"), bool) or not isinstance(d.get("response"), str):
log_warning(f"Unparseable answer attempt for section {section_ref!r}; treating as 'cannot answer'.")
return AnswerAttempt(can_answer=False, response="Could not extract a usable answer from this section.")
return AnswerAttempt(can_answer=d["can_answer"], response=d["response"])
```

`can_answer=False` just sends the loop to the next section, and the failed iteration
stays visible in the trace.

About 6 lines plus a test that feeds junk through a mock session. Happy to open the PR.

PS: found this while working on #25.

Contributor guide

Open the contributing guide

Research direction

Start in docling_agent/agent/rag.py with _attempt_answer and compare its find_json_dicts handling with _select_section. Add a test that feeds unparseable output through a mock session and verify the RAG loop continues with a cannot-answer attempt rather than crashing. Done means junk output no longer raises IndexError or KeyError and the failed iteration remains visible in the trace.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
88/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.