Automattic / Automattic/harper

Non-breaking space (U+00A0) breaks all multi-token lints; tokenizer only treats ASCII space as whitespace

Open Beginner friendly
#3,863 1 comment 2 reactions 0 assignees View on GitHub
bug harper-core linting
Dominant language
Rust
Stars
15.4k
Forks
627
Avg merge
1d 15h
Merged PRs (30d)
106

Description

## Summary

`lex_spaces` in `harper-core/src/lexing/mod.rs` only matches the literal ASCII space `' '` (U+0020). A non-breaking space (U+00A0) is never lexed as a `Space` token, so any lint requiring cross-word context (repeated words, a/an agreement, that/than, Oxford comma, unit expansion, etc.) silently produces nothing, while single-word lints (spelling, typos) still fire.

This hits the Chrome extension hard on Gmail: Gmail's compose contenteditable stores inter-word spaces as U+00A0, so Harper appears to "half work" there, spelling lints show, grammar lints never do. Any rich editor that uses nbsp is affected the same way.

## Reproduction (engine only, no browser needed)

Node 22, `harper.js` 2.4.0 from npm:

```js
import { LocalLinter } from 'harper.js';
import { binaryInlined as binary } from 'harper.js/binaryInlined';

const linter = new LocalLinter({ binary });
const text = `There are some cases where the the standard grammar checkers don't cut it. That;s where Harper comes in handy.`;

for (const [name, t] of [['U+0020', text], ['U+00A0', text.replaceAll(' ', '\u00A0')]]) {
const lints = await linter.lint(t, { language: 'plaintext' });
console.log(name, lints.map(l => l.lint_kind_pretty()));
}
```

Output:

```
U+0020 [ 'Repetition', 'Typo' ]
U+00A0 [ 'Typo' ]
```

The `Repetition` lint on "the the" disappears when the separator is U+00A0. The same pattern kills `an language` (agreement), `faster that` (that/than), `10 ms` (unit expansion), and Oxford comma detection.

## Reproduction (Chrome extension on Gmail)

1. Install the Chrome extension (v2.6.0) in any Chromium browser, Gmail is on the default-enabled domain list.
2. Open a Gmail compose window and paste the sample text from the writewithharper.com demo.
3. The demo editor reports 13 problems. In Gmail only the ~6 single-word spelling/typo lints render; "the the", "an language", "emails" (Oxford comma), "not online", "ms", "faster that" are all missing.
4. Confirm the cause in DevTools on the Gmail tab:
`(document.querySelector('div[aria-label="Message Body"]').textContent.match(/\u00A0/g) || []).length` returns > 0.
5. Replacing the U+00A0 characters with regular spaces in the compose DOM makes all 13 lints appear.

## Root cause

`lex_spaces` counts only `' '`:

```rust
fn lex_spaces(source: &[char]) -> Option {
let count = source.iter().take_while(|c| **c == ' ').count();
...
}
```

`\u{00A0}` appears nowhere in harper-core, so nbsp falls through and severs the token stream at that point.

## Suggested fix

Either treat U+00A0 (and ideally the other Unicode space separators: U+202F, U+2007, U+2009, etc.) as a `Space` token in the lexer, or normalize them during text extraction in the browser extension / lint-framework. The lexer fix benefits every integration at once.

Environment: Linux, Chromium-based browser, Harper Chrome extension 2.6.0, harper.js 2.4.0, repo at current master.

---

NOTE: This has been tested, and I've *manually* confirmed this is the issue by double-checking the work myself. However, this issue was originally identified using Claude Fable 5 - high thinking.

EDIT: Please note this is also happening in WhatsApp and by the looks of it any website that encodes characters this way.

Contributor guide

Open the contributing guide

Research direction

Start in harper-core/src/lexing/mod.rs at lex_spaces, then run the Node 22 harper.js reproduction comparing U+0020 with U+00A0. Trace how the lexer emits tokens for each separator and verify that cross-word lints such as Repetition return results for non-breaking spaces without losing the existing ASCII-space behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
tooling
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.