eellak / eellak/glossAPI

Narrow URL-like protection so PostScript glyph names inside numeric ranges are cleaned

Open
#100 1 comment 0 reactions 0 assignees View on GitHub
cleaning tokenizer-followup
Dominant language
Python
Stars
141
Forks
41
PR merge metrics
No merged PRs in 30d

Description

During the 2026-04-28 wave-3 tokenizer review, the C1 added vocab had one residual PostScript-name token: `/hyphenminus`.

Corpus spot checks showed contexts such as:

- `4.600/hyphenminus5.600 KDa`
- `0.3/hyphenminus 100%`
- `50.000/hyphenminus250.000 κατοίκους`

The cleaner strips standalone `foo /hyphenminus bar`, and correctly preserves URL/path contexts such as `https://example.org/a/hyphenminus/b`, but it currently preserves numeric slash ranges because `URL_LIKE_TOKEN_REGEX` treats tokens like `4.600/hyphenminus5.600` as URL-like host/path tokens.

Suggested later fix:

- keep protecting real URLs, `www.` links, and host/path tokens with alphabetic host components
- do not classify purely numeric dotted ranges followed by slash as URL-like
- add tests for numeric `/hyphenminus` stripping and real URL preservation

This is a small cleanup follow-up, not a blocker for the completed tokenizer wave.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.