Automattic / Automattic/harper
Non-breaking space (U+00A0) breaks all multi-token lints; tokenizer only treats ASCII space as whitespace
- Dominant language
- Rust
- Stars
- 15.4k
- Forks
- 627
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 106
Description
## Summary
`lex_spaces` in `harper-core/src/lexing/mod.rs` only matches the literal ASCII space `' '` (U+0020). A non-breaking space (U+00A0) is never lexed as a `Space` token, so any lint requiring cross-word context (repeated words, a/an agreement, that/than, Oxford comma, unit expansion, etc.) silently produces nothing, while single-word lints (spelling, typos) still fire.
This hits the Chrome extension hard on Gmail: Gmail's compose contenteditable stores inter-word spaces as U+00A0, so Harper appears to "half work" there, spelling lints show, grammar lints never do. Any rich editor that uses nbsp is affected the same way.
## Reproduction (engine only, no browser needed)
Node 22, `harper.js` 2.4.0 from npm:
```js
import { LocalLinter } from 'harper.js';
import { binaryInlined as binary } from 'harper.js/binaryInlined';
const linter = new LocalLinter({ binary });
const text = `There are some cases where the the standard grammar checkers don't cut it. That;s where Harper comes in handy.`;
for (const [name, t] of [['U+0020', text], ['U+00A0', text.replaceAll(' ', '\u00A0')]]) {
const lints = await linter.lint(t, { language: 'plaintext' });
console.log(name, lints.map(l => l.lint_kind_pretty()));
}
```
Output:
```
U+0020 [ 'Repetition', 'Typo' ]
U+00A0 [ 'Typo' ]
```
The `Repetition` lint on "the the" disappears when the separator is U+00A0. The same pattern kills `an language` (agreement), `faster that` (that/than), `10 ms` (unit expansion), and Oxford comma detection.
## Reproduction (Chrome extension on Gmail)
1. Install the Chrome extension (v2.6.0) in any Chromium browser, Gmail is on the default-enabled domain list.
2. Open a Gmail compose window and paste the sample text from the writewithharper.com demo.
3. The demo editor reports 13 problems. In Gmail only the ~6 single-word spelling/typo lints render; "the the", "an language", "emails" (Oxford comma), "not online", "ms", "faster that" are all missing.
4. Confirm the cause in DevTools on the Gmail tab:
`(document.querySelector('div[aria-label="Message Body"]').textContent.match(/\u00A0/g) || []).length` returns > 0.
5. Replacing the U+00A0 characters with regular spaces in the compose DOM makes all 13 lints appear.
## Root cause
`lex_spaces` counts only `' '`:
```rust
fn lex_spaces(source: &[char]) -> Option {
let count = source.iter().take_while(|c| **c == ' ').count();
...
}
```
`\u{00A0}` appears nowhere in harper-core, so nbsp falls through and severs the token stream at that point.
## Suggested fix
Either treat U+00A0 (and ideally the other Unicode space separators: U+202F, U+2007, U+2009, etc.) as a `Space` token in the lexer, or normalize them during text extraction in the browser extension / lint-framework. The lexer fix benefits every integration at once.
Environment: Linux, Chromium-based browser, Harper Chrome extension 2.6.0, harper.js 2.4.0, repo at current master.
---
NOTE: This has been tested, and I've *manually* confirmed this is the issue by double-checking the work myself. However, this issue was originally identified using Claude Fable 5 - high thinking.
EDIT: Please note this is also happening in WhatsApp and by the looks of it any website that encodes characters this way.
Contributor guide
Research direction
Start in harper-core/src/lexing/mod.rs at lex_spaces, then run the Node 22 harper.js reproduction comparing U+0020 with U+00A0. Trace how the lexer emits tokens for each separator and verify that cross-word lints such as Repetition return results for non-breaking spaces without losing the existing ASCII-space behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- tooling
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100