docling-project / docling-project/docling-agent
RAG: _attempt_answer crashes on unparseable model output
- Dominant language
- Python
- Stars
- 166
- Forks
- 21
- PR merge metrics
- No merged PRs in 30d
Description
`_attempt_answer` in `docling_agent/agent/rag.py` parses the model response without any check:
```python
d = find_json_dicts(answer)[0]
return AnswerAttempt(can_answer=d["can_answer"], response=d["response"])
```
`_select_section`, in the same file, guards the exact same pattern and falls back to
picking the first unvisited section. The other `find_json_dicts` call sites
(orchestrator, enricher, editor) all check the result too. This is the only
unguarded one.
It can trigger on every backend. On the mellea backend the retry budget is best
effort: when rejection sampling runs out, the session returns a failed sample
instead of raising. On the direct HTTP backends (Ollama and the OpenAI-compatible
ones), requirements are not enforced at all, so the first response without a valid
JSON block reaches the parse directly.
Either way this line raises `IndexError` (no JSON found) or `KeyError` (missing key)
and the whole RAG run aborts. Without the crash, `_rag_loop` would have moved on to
the next section, or returned the partial answer it already had.
Fix, mirroring `_select_section`:
```python
dicts = find_json_dicts(answer)
d = dicts[0] if dicts else {}
if not isinstance(d.get("can_answer"), bool) or not isinstance(d.get("response"), str):
log_warning(f"Unparseable answer attempt for section {section_ref!r}; treating as 'cannot answer'.")
return AnswerAttempt(can_answer=False, response="Could not extract a usable answer from this section.")
return AnswerAttempt(can_answer=d["can_answer"], response=d["response"])
```
`can_answer=False` just sends the loop to the next section, and the failed iteration
stays visible in the trace.
About 6 lines plus a test that feeds junk through a mock session. Happy to open the PR.
PS: found this while working on #25.
Contributor guide
Research direction
Start in docling_agent/agent/rag.py with _attempt_answer and compare its find_json_dicts handling with _select_section. Add a test that feeds unparseable output through a mock session and verify the RAG loop continues with a cannot-answer attempt rather than crashing. Done means junk output no longer raises IndexError or KeyError and the failed iteration remains visible in the trace.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 88/100