design(flows): structure tree drafted from truncated page text (1200 chars), diverging from PageIndex full-page construction

@keli-wen is already working on this.

Since Jul 23, 2026.

## Summary

QuantMind's paper structure tree (the "PageIndex-style" `PaperStructureTree` produced by `PaperFlow(PaperStructureCfg).build`) is drafted by a model that only ever sees, per page, the first `page_text_chars` characters (default `1200`) plus the deterministically-extracted headings. The reference implementation this feature is modeled on — [VectifyAI/PageIndex](https://github.com/VectifyAI/PageIndex) — never truncates page bodies: it feeds the model each page's full text (batched into token-bounded windows) and verifies titles against real page content. This issue records the divergence and the resulting shortcomings as a future development point. It is not a blocker for current functionality (node content itself is filled from full page text), and it is unrelated to the usage-tracking work that surfaced it.

## How QuantMind builds the tree today

Three stages (see `contexts/design/mind/retrieval.md`):

1. **Deterministic outline signals** — `extract_outline_signals` (`quantmind/preprocess/outline.py`) scans the whole document and emits all heading candidates + table-of-contents pages + printed-to-physical page offset. Full, not truncated.
2. **Model draft structuring** — `quantmind/flows/paper/_structure.py` sends the model `{outline: , pages: [{page_number, text: page.text[:page_text_chars]}]}` and gets back the tree skeleton (title + inclusive page span + one-line summary per node, plus a quality rating). This is the only stage that truncates, at `page.text[: cfg.page_text_chars]`, with `page_text_chars` defaulting to `1200` (`quantmind/configs/structure.py`, alongside `max_depth=6`, `max_nodes=128`).
3. **Deterministic content fill** — `PaperStructureTree.from_draft` (`quantmind/knowledge/paper.py`) populates each leaf node's `content` by joining the full text of its cited pages. Full text.

So the model decides which pages belong to which section, and writes each node's summary, from headings + the first ~1200 chars (~200-300 words, often under half a dense page) of each page — in a single truncated pass.

## How official PageIndex builds the tree

From the source (`pageindex/page_index.py`) and docs:

- **Full page text, windowed by page count, never truncated.** `page_list_to_group_text(..., max_tokens=20000, overlap_page=1)` concatenates each page's complete text into ~20k-token windows with a 1-page overlap; there is no per-page character/token slicing anywhere in the source.
- **TOC-first, generate as fallback.** `find_toc_pages` detects a table of contents in the first ~20 pages; `process_toc_with_page_numbers` maps printed page numbers to physical indices, `process_toc_no_page_numbers` fuzzy-matches titles to pages; when there is no TOC (the usual case for arXiv papers), `process_no_toc` → `generate_toc_init` / `generate_toc_continue` infer the structure from full page-text windows.
- **Depth from the document's own nesting plus recursive splitting** of oversized nodes (`max-pages-per-node` ≈ 10, `max-tokens-per-node` ≈ 20000), not a fixed max depth.
- **Title verification against real content.** `check_title_appearance` confirms each extracted title actually appears on its assigned page; low accuracy triggers retry / a more conservative fallback.
- **Summaries in a dedicated pass over full section text.** `add_node_text` attaches each node's section text from its page range, then `generate_summaries_for_structure` summarizes from that full text — not from a truncated draft view.

## The divergence, and why it is a quality risk

- **Structure fidelity.** Deciding page spans from headings + the first 1200 chars breaks when a section starts in the lower part of a dense page, when the deterministic heading extractor misses a heading, or when structure must be inferred from body prose. A wrong page span then feeds the wrong pages into stage 3's content fill.
- **Summary fidelity.** Node summaries are written from the same truncated view, so they can describe only the first portion of a section; PageIndex writes summaries from full section text in a separate pass.
- **It only looks fine on sparse inputs.** The golden test fixture has ~800 chars/page, so truncation never triggers there — the gap is invisible on that fixture and appears on real, dense papers. (A structure build on the fixture reports ~1.1-1.5k input tokens precisely because the whole prompt is instructions + outline + four short pages.)

Content is not truncated (stage 3 uses full page text), so this is a **structure / summary fidelity gap, not a content-truncation bug**.

## Possible future directions

- **Feed full page text in token-bounded windows** (PageIndex's approach): bound the prompt by pages-per-window (e.g. ~20k tokens, 1-page overlap), not by clipping each page. Cap cost by page count, not by character truncation.
- **Add a title-verification pass** (`check_title_appearance`-style) and, for documents that have one, TOC detection with printed→physical page mapping.
- **Generate node summaries in a dedicated pass over full section text** (stage 3 already reads it) rather than during the truncated draft.
- **At minimum, raise the `page_text_chars` default** so a typical page is not clipped, and document when truncation is safe (sparse pages / clean TOC).

## References

- QuantMind: `quantmind/flows/paper/_structure.py`, `quantmind/configs/structure.py`, `quantmind/preprocess/outline.py`, `quantmind/knowledge/paper.py` (`from_draft`), `contexts/design/mind/retrieval.md`.
- PageIndex: , source `pageindex/page_index.py`, and the [document-processing flow](https://deepwiki.com/VectifyAI/PageIndex/2.2-document-processing-flow).