--semantic never breaks a sentence that ends in a code span or emphasis (SENTENCE_END_RE delimiter class)
- Dominant language
- Python
- Stars
- 81
- Forks
- 11
- Avg merge
- 10h 53m
- Merged PRs (30d)
- 6
Description
__Claude wrote this but I reviewed it and I'm the one posting it__
## Summary
`--semantic` does not break at a sentence boundary when the terminating `.`/`?`/`!` directly follows
an inline code span, bold, italic, or strikethrough. A sentence ending in a plain word, a quoted
string, or a parenthetical breaks correctly, so the failing cases are specifically Markdown markup
delimiters sitting between the last letter and the terminator.
This bites hard in technical prose, where sentences routinely end on a backticked identifier or
filename. In one repo of mine it accounts for 180 unbroken sentence boundaries across 56 of 96
Markdown files — most of the value of `--semantic`.
## Root cause
`flowmark/linewrapping/sentence_split_regex.py`:
```python
SENTENCE_END_RE = regex.compile(r"(\b\p{L}+[\p{Ll}])([.?!]['\"’”)]?|['\"’”)][.?!]) *$")
```
The pattern requires the terminator to sit immediately after a letter, optionally separated by one
character from the class `['\"’”)]`. That class covers quotes and a closing paren but not the
Markdown emphasis delimiters `` ` ``, `*`, `_`, or `~`, so `` `token`. `` never matches and the
splitter treats the sentence as unfinished.
Confirmed directly against the heuristic:
```python
from flowmark.linewrapping.sentence_split_regex import heuristic_end_of_sentence as h
h("ends with word.") # True
h('ends with "quote".') # True
h("ends with (paren).") # True
h("ends with `code`.") # False
h("ends with **bold**.") # False
h("ends with *ital*.") # False
h("ends with _ital_.") # False
h("ends with ~~strike~~.") # False
h("ends with `code`?") # False
h("ends with `code`!") # False
```
Adding `` ` ``, `*`, `_`, and `~` to that character class looks like it would cover this, though
multi-character delimiters (`**`, `~~`) need more than a single optional character, so the
`[...]?` may need to become a quantified group.
## Reproduction
At the API level:
```python
from flowmark import split_sentences_regex
S = " Then a second sentence of adequate length here. And a third sentence to be sure."
split_sentences_regex("The system uses a token." + S)
# ['The system uses a token.',
# 'Then a second sentence of adequate length here.',
# 'And a third sentence to be sure.']
split_sentences_regex("The system uses a `token`." + S)
# ['The system uses a `token`. Then a second sentence of adequate length here.',
# 'And a third sentence to be sure.']
```
End to end, with `reformat_text(..., semantic=True)` or `flowmark --semantic --width 100000`:
| Token before the period | First sentence | Breaks? |
| --- | --- | --- |
| plain word | `The system uses a token.` | ✅ |
| code span | ``The system uses a `token`.`` | ❌ |
| bold | `The system uses a **token**.` | ❌ |
| italic | `The system uses a *token*.` | ❌ |
| link | `The system uses a [token](t.md).` | ✅ |
| quoted string | `The system uses a "token".` | ✅ |
| code span, then a word | ``The system uses a `token` here.`` | ✅ |
| code span, question mark | ``Does the system use a `token`?`` | ❌ |
Actual for the code-span case:
```markdown
The system uses a `token`. Then a second sentence of adequate length here.
And a third sentence to be sure.
```
Expected:
```markdown
The system uses a `token`.
Then a second sentence of adequate length here.
And a third sentence to be sure.
```
Note the second boundary does break, so this is not a whole-paragraph failure — only the boundary
preceded by the code span is skipped.
## One repro gotcha
`SENTENCE_MIN_LENGTH = 15` means a short two-sentence line such as `One two. Three four.` stays on
one line regardless of `--width`. My first attempt at a minimal repro was misleading for this
reason. Every example above is padded well past that threshold, and the plain-word control breaks at
the same length.
## Versions
- `flowmark v0.7.3` (Python), via `uvx --from flowmark flowmark`, Linux x86_64.
The Rust port reproduces the same matrix identically (`flowmark 0.3.2`, base v0.3.2), which fits the
root cause: the two use different Markdown parsers (marko vs comrak) but share this sentence-split
heuristic. Worth a case in the parity corpus from flowmark-rs#60.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in flowmark/linewrapping/sentence_split_regex.py at SENTENCE_END_RE and heuristic_end_of_sentence, then reproduce the documented split_sentences_regex cases for code, emphasis, and strikethrough delimiters. Ensure sentence boundaries before ., ?, and ! are recognized without regressing the existing word, quote, parenthetical, and link cases; consider the parity corpus noted in flowmark-rs#60.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- tooling
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 74/100