Support incremental tokenization
Nobody has claimed this yet.
- Dominant language
- JavaScript
- Stars
- 13k
- Forks
- 1.4k
- Avg merge
- 15h 36m
- Merged PRs (30d)
- 3
Description
Motivation
Without incremental tokenization the syntax highlighter becomes essentially too inefficient to be used in a code editor, for example.
Description
Incremental tokenization allows for pausing and resuming tokenization, allowing for tokenizing a string line by line, and ideally also allowing to restart tokenization from a point different from the very start (after the previous lines have been tokenized already).
Without this feature syntax highlighting would probably be too expensive to perform in one go and/or visibly blocking for the user.
Alternatives
Using vscode-textmate, but that's kinda huge (and slow) since it needs Oniguruma to work.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names no files, tests, or entry points. Start by locating Prism's tokenization entry point and related tests; define how tokenization can pause and resume line by line, and how restarting after previously tokenized lines should work, then verify that syntax highlighting avoids blocking on a full document.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript
- Domain
- tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100