jlevy / jlevy/flowmark

--semantic never breaks a sentence that ends in a code span or emphasis (SENTENCE_END_RE delimiter class)

Open
#68 2 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
81
Forks
11
Avg merge
10h 53m
Merged PRs (30d)
6

Description

__Claude wrote this but I reviewed it and I'm the one posting it__

## Summary

`--semantic` does not break at a sentence boundary when the terminating `.`/`?`/`!` directly follows
an inline code span, bold, italic, or strikethrough. A sentence ending in a plain word, a quoted
string, or a parenthetical breaks correctly, so the failing cases are specifically Markdown markup
delimiters sitting between the last letter and the terminator.

This bites hard in technical prose, where sentences routinely end on a backticked identifier or
filename. In one repo of mine it accounts for 180 unbroken sentence boundaries across 56 of 96
Markdown files — most of the value of `--semantic`.

## Root cause

`flowmark/linewrapping/sentence_split_regex.py`:

```python
SENTENCE_END_RE = regex.compile(r"(\b\p{L}+[\p{Ll}])([.?!]['\"’”)]?|['\"’”)][.?!]) *$")
```

The pattern requires the terminator to sit immediately after a letter, optionally separated by one
character from the class `['\"’”)]`. That class covers quotes and a closing paren but not the
Markdown emphasis delimiters `` ` ``, `*`, `_`, or `~`, so `` `token`. `` never matches and the
splitter treats the sentence as unfinished.

Confirmed directly against the heuristic:

```python
from flowmark.linewrapping.sentence_split_regex import heuristic_end_of_sentence as h

h("ends with word.") # True
h('ends with "quote".') # True
h("ends with (paren).") # True
h("ends with `code`.") # False
h("ends with **bold**.") # False
h("ends with *ital*.") # False
h("ends with _ital_.") # False
h("ends with ~~strike~~.") # False
h("ends with `code`?") # False
h("ends with `code`!") # False
```

Adding `` ` ``, `*`, `_`, and `~` to that character class looks like it would cover this, though
multi-character delimiters (`**`, `~~`) need more than a single optional character, so the
`[...]?` may need to become a quantified group.

## Reproduction

At the API level:

```python
from flowmark import split_sentences_regex

S = " Then a second sentence of adequate length here. And a third sentence to be sure."

split_sentences_regex("The system uses a token." + S)
# ['The system uses a token.',
# 'Then a second sentence of adequate length here.',
# 'And a third sentence to be sure.']

split_sentences_regex("The system uses a `token`." + S)
# ['The system uses a `token`. Then a second sentence of adequate length here.',
# 'And a third sentence to be sure.']
```

End to end, with `reformat_text(..., semantic=True)` or `flowmark --semantic --width 100000`:

| Token before the period | First sentence | Breaks? |
| --- | --- | --- |
| plain word | `The system uses a token.` | ✅ |
| code span | ``The system uses a `token`.`` | ❌ |
| bold | `The system uses a **token**.` | ❌ |
| italic | `The system uses a *token*.` | ❌ |
| link | `The system uses a [token](t.md).` | ✅ |
| quoted string | `The system uses a "token".` | ✅ |
| code span, then a word | ``The system uses a `token` here.`` | ✅ |
| code span, question mark | ``Does the system use a `token`?`` | ❌ |

Actual for the code-span case:

```markdown
The system uses a `token`. Then a second sentence of adequate length here.
And a third sentence to be sure.
```

Expected:

```markdown
The system uses a `token`.
Then a second sentence of adequate length here.
And a third sentence to be sure.
```

Note the second boundary does break, so this is not a whole-paragraph failure — only the boundary
preceded by the code span is skipped.

## One repro gotcha

`SENTENCE_MIN_LENGTH = 15` means a short two-sentence line such as `One two. Three four.` stays on
one line regardless of `--width`. My first attempt at a minimal repro was misleading for this
reason. Every example above is padded well past that threshold, and the plain-word control breaks at
the same length.

## Versions

- `flowmark v0.7.3` (Python), via `uvx --from flowmark flowmark`, Linux x86_64.

The Rust port reproduces the same matrix identically (`flowmark 0.3.2`, base v0.3.2), which fits the
root cause: the two use different Markdown parsers (marko vs comrak) but share this sentence-split
heuristic. Worth a case in the parity corpus from flowmark-rs#60.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in flowmark/linewrapping/sentence_split_regex.py at SENTENCE_END_RE and heuristic_end_of_sentence, then reproduce the documented split_sentences_regex cases for code, emphasis, and strikethrough delimiters. Ensure sentence boundaries before ., ?, and ! are recognized without regressing the existing word, quote, parenthetical, and link cases; consider the parity corpus noted in flowmark-rs#60.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.