jlevy / jlevy/flexdoc

pprose (practical-prose) use case: document-model needs and a few gaps after the 0.1.0 migration

Open
#5 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1
Forks
0
PR merge metrics
No merged PRs in 30d

Description

## Context: the practical-prose / pprose use case

[practical-prose](https://github.com/jlevy/practical-prose) ships **pprose**, an
evaluation toolkit for practical documents. It computes a structural + linguistic
metrics report over Markdown (`pprose metrics`), and is building a two-phase AI-prose
linter that flags constructions by exact source span and verifies each candidate with a
focused model call.

pprose just migrated from `chopdiff==0.3.1` to `flexdoc==0.1.0` and dropped chopdiff
entirely (it only ever used the document model, not the diff/windowed-transform layer).
The migration was a clean drop-in: `chopdiff.docs.TextDoc` -> `flexdoc.FlexDoc`,
identical word/sentence/paragraph/line counts, and the fixture-locked metrics test passed
byte-for-byte. Thank you for the clean extraction.

This issue records the full set of document-model needs pprose has, so flexdoc can be a
complete foundation for "comprehensive review and control of practical documents." Most
of it already works in 0.1.0; the genuine gaps are a short, prioritized list at the end.

## What already works well in 0.1.0 (no action needed)

Verified empirically against the published 0.1.0 wheel:

- **Prose-only sizing** via `FlexDoc.filtered(include={...}).size(TextUnit.{words,sentences,paragraphs})`.
- **Frontmatter isolation** (`FlexDoc.frontmatter`; excluded from `paragraphs` and all counts).
- **Typed block metadata**: `Block.code_info` (language, line_count), `Block.table_info`
(rows, cols, cells, alignments), `Block.list_info` (ordered, start, max_depth, item_count).
- **Sections + rollups**: `sections()`, `Section.size(unit, subtree=)`, `section_size_tree()`, `toc()`.
- **Inline node collection** including **footnote references**: `collect(kinds={NodeKind.footnote_ref | link | image | code_span}, recursive=True)` returns typed nodes with exact spans and useful attrs (`url`, `text`, `label`, `sentence`, `section`).
- **Exact offsets + inversion**: `Offsets` on every paragraph/sentence/inline node; `paragraph_at_offset`, `sentence_at_offset`.
- **`SpanRef`** quote-canonical anchoring for span-stable annotations.

Notably, this **closes all five** of the old chopdiff feature requests
(jlevy/chopdiff#18-#22), including `NodeKind.footnote_ref`, which is implemented in 0.1.0.

## Requests (prioritized)

### 1. Typed link *form* discriminator (highest value)

pprose reports links broken down by Markdown form: **inline** `[t](u)`, **autolink**
``, **bare URL** `https://...`, **reference-use** `[t][id]`, **reference-definition**
`[id]: url`, and **image** `![t](u)`. In 0.1.0 every one of these (except image) collapses
to `NodeKind.link` with attrs `{url, text, sentence}` and no way to tell them apart, and
**reference definitions are not surfaced at all** (resolved into the use's `url`, no node
or count). A bare URL is also returned as a `link` (GFM autolinking), which conflicts with
pprose's "bare URL = not in link syntax" metric.

Request: a typed `form` on link nodes / `Link`, e.g.
`Literal["inline","autolink","bare_url","reference","image"]`, so consumers can count by
form without heuristics (today the only signal is `text == url`, which cannot separate
autolink from bare URL, nor reference-use from inline).

### 2. Surface link reference definitions

Expose `[id]: url` definition lines, either as a node kind (e.g. `link_ref_def`) or a
count/list on `FlexDoc`. pprose currently regexes these for its `links_reference_definitions`
metric; flexdoc resolves them away with no typed handle.

### 3. `block_at_offset` (offset -> block inversion)

The linter maps a match offset back to its enclosing block to decide skip rules
(code/table/frontmatter) and build context windows. `paragraph_at_offset` and
`sentence_at_offset` exist; a `block_at_offset` (or a documented "use
`collect(overlaps=(o, o+1), layer={Layer.markdown})`" recipe) would complete the set.
Low priority since it is derivable.

### 4. Fuzzy / edit-distance `SpanRef` re-anchoring (forward-looking)

Already on the roadmap (spec §14). The two-phase linter applies corrections and re-runs;
corrections must re-anchor after the text around them shifts. A +1 with this concrete use
case.

## Ergonomics note (not a bug)

Inline collection requires `recursive=True`. `collect(kinds={NodeKind.link}, inline=True)`
alone returns `[]` because, with no tree/interval relation, candidates default to root
(block) nodes and inline nodes are nested. This is easy to trip over; a one-line note in
`collect()`'s docstring or `usage.md` (or having explicit inline kinds imply `recursive`)
would save the next consumer the same detour.

## Out of scope for flexdoc (recorded for completeness)

pprose's lint-pattern metrics (banned register, spaced-em-dash discipline,
replacement-history / pedantic-marker phrases, ALL-CAPS bracket tags, generic headings)
are editorial heuristics, not document structure. pprose will keep running these as
regexes over the prose-only `filtered(...).reassemble()` text; flexdoc need not type them.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.