jlevy / jlevy/flexdoc

Configurable size-counting scopes (prose-only, with/without tables/inline-code) without re-parsing

Open
#8 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1
Forks
0
PR merge metrics
No merged PRs in 30d

Description

## Context

[pprose](https://github.com/jlevy/practical-prose) computes document metrics and just migrated its sizing onto flexdoc 0.2.0 (thanks!). In doing so we hit a counting-scope gap: there's no clean, configurable way to get **size counts (words / sentences / paragraphs / lines) for a specific prose scope**, and the scope you want genuinely differs by use case. Filing as a feature request; not blocking (we have a working approach), but it'd let consumers stop composing ad-hoc projections.

## The problem

We want size counts over the *same* prose scope our lint runs over (so `em_dash_density = em_dashes / words` has a consistent numerator and denominator). The natural prose projection is `prose_text()`, but the available options each fall short:

- **`filtered(include={paragraph, list, ...}).size()`** sizes a real sub-document (good, single parse), but the scope is fixed by which `BlockType`s you include: it excludes heading and table text, and it *keeps* inline-code spans inside paragraphs (so `` `code` `` tokens count as words). That's the wrong scope for our case and not adjustable along the axes we care about.
- **`prose_text(include_tables=…)`** is exactly the scope we want (drops inline code, unwraps links/images, excludes frontmatter + code blocks, tables optional) — but it returns a **string**, so to get sizes we have to **re-parse** it: `FlexDoc.from_text(doc.prose_text(...)).size(...)`. That's a second parse of derived text, and a bit circular.

So today the choices are "a sub-document with the wrong scope" or "the right scope but only as text, requiring a re-parse."

## What would help (not prescriptive)

A way to get **sizes for a prose projection without re-parsing**, with the scope **configurable per use case**. Concretely, the axes that matter to us:

- **Tables in or out.** Word counts *with* and *without* table content are both legitimately useful (table-cell text is content for some metrics, noise for prose-density ratios). We'd like to ask for either.
- **Inline code in or out.** For prose word counts, inline-code tokens are usually noise; for other uses they're content.
- **Headings in or out**, similarly.

Whatever shape fits flexdoc best — e.g. a `size(unit, scope=…)` with named scopes, options on `filtered()`/`prose_text()` that also expose `.size()`, or a small `ProseScope` config — so a consumer can say "words, prose-only, without tables" and get a consistent answer without re-parsing or hand-assembling `BlockType` sets.

## What "done" looks like for us

- Get words / sentences / paragraphs / lines for a named/parameterized prose scope in one call, no re-parse.
- Select table inclusion (and ideally inline-code / heading inclusion) per call.
- The scope used for sizing is the same one `prose_text()` exposes, so text-based lint and numeric sizes stay consistent.

## Current pprose workaround

We size over `FlexDoc.from_text(doc.prose_text(include_tables=True)).size(...)` and run lint over the same `prose_text(...)` string, which keeps the two consistent. It works; the re-parse and the fixed scope are the rough edges this request would smooth.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.