LLMQuant / LLMQuant/quant-mind

paper_flow() fails on its own README example: dict-typed nodes field breaks strict structured output, UUID id fields reject the model's natural output

Open
#91 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area: flows stale type: bug
Dominant language
Python
Stars
3k
Forks
484
PR merge metrics
No merged PRs in 30d

Description

Summary

The single-paper example in the README does not run against the currently resolved dependency versions (openai-agents>=0.14, no ceiling, resolves to 0.18.1):

import asyncio
from quantmind.configs import PaperFlowCfg
from quantmind.configs.paper import ArxivIdentifier
from quantmind.flows import paper_flow

async def main() -> None:
    paper = await paper_flow(
        ArxivIdentifier(id="2401.12345"),
        cfg=PaperFlowCfg(model="gpt-4o-mini"),
    )

asyncio.run(main())

Two separate bugs stack on top of each other.

Bug 1 — Agent(output_type=Paper) raises before any LLM call

TreeKnowledge.nodes is dict[UUID, TreeNode] (quantmind/knowledge/_tree.py). OpenAI's strict-mode structured output — the default in current openai-agents releases — cannot represent dict-typed fields at all (arbitrary keys are incompatible with a fixed JSON Schema). paper_flow (quantmind/flows/paper.py) passes the bare Paper type straight into Agent(output_type=out_type), with no strict_json_schema=False escape hatch, so the run fails immediately:

agents.exceptions.UserError: Strict JSON schema is enabled, but the output type is not valid.
Either make the output type strict, or wrap your type with AgentOutputSchema(YourType, strict_json_schema=False)

Bug 2 — once strict mode is off, UUID-typed id fields reject the model's output

Wrapping the output type in AgentOutputSchema(Paper, strict_json_schema=False) gets past bug 1, but a non-strict schema drops the UUID format constraint. The extraction agent then does exactly what you'd expect an LLM to do when free-form: it fills TreeNode.node_id / parent_id / children_ids with readable slugs ("root", "introduction", "methodology") and BaseKnowledge.id with the paper's arXiv id ("2404.11584"), rather than leaving these at their UUID4 defaults. Validation then fails with ~80+ uuid_parsing errors.

Repro

Ran the unmodified README example against a real paper (arXiv 2404.11584, "The landscape of emerging AI agent architectures for reasoning, planning, and tool calling: A survey", model gpt-4o-mini) on the current main (8e21888). Fails both ways described above. Full traceback available on request.

Fix

Opened # — changes TreeNode.node_id/parent_id/children_ids, TreeKnowledge.root_node_id/nodes, BaseKnowledge.id, and PaperKnowledgeCard.paper_id from UUID to str (accepts both UUIDs and readable slugs), and has paper_flow pass AgentOutputSchema(out_type, strict_json_schema=False) instead of a bare type. Verified: the unmodified README example now completes end-to-end with zero validation errors against the same real paper, and the full test suite passes (232/233 — the one failure is pre-existing and unrelated, a Windows path-separator assertion in a local-file test).

Environment

  • openai-agents==0.18.1 (resolved from the unpinned >=0.14 floor)
  • Windows 11, Python 3.12.12

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with quantmind/flows/paper.py and quantmind/knowledge/_tree.py, then inspect the Paper and related knowledge models named in the issue. Run the README single-paper example against openai-agents 0.18.1 and the full test suite. Done means the example completes without schema or UUID validation errors and existing tests remain passing aside from the noted unrelated failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.