microsoft / microsoft/TypeScript-TmLanguage
[JavaScript] Tokenization breaks when new line starts with less-than operator
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 471
- Forks
- 149
- PR merge metrics
- No merged PRs in 30d
Description
Does this issue occur when all extensions are disabled?: Yes
VS Code version: Code 1.66.0 (e18005f0f1b33c29e81d732535d8c0e47cafb0b5, 2022-03-30T05:50:41.156Z)
OS version: Linux x64 4.15.0-175-generic
Restricted Mode: No
System Info
| Item | Value |
|---|---|
| CPUs | Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz (8 x 3541) |
| GPU Status | 2d_canvas: enabled canvas_oop_rasterization: disabled_off direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on oop_rasterization: disabled_off opengl: enabled_on rasterization: disabled_software raw_draw: disabled_off_ok skia_renderer: enabled_on video_decode: disabled_software video_encode: disabled_software vulkan: disabled_off webgl: enabled webgl2: enabled |
| Load (avg) | 0, 0, 1 |
| Memory (System) | 31.26GB (23.84GB free) |
| Process Argv | --unity-launch --crash-reporter-id c990702e-9bfc-48d9-b2b9-78bab7fdb823 |
| Screen Reader | no |
| VM | 0% |
| DESKTOP_SESSION | ubuntu |
| XDG_CURRENT_DESKTOP | Unity |
| XDG_SESSION_DESKTOP | ubuntu |
|XDG_SESSION_TYPE|x11|
Simple example
- In a new editor, set the language mode to plain
JavaScript - Type the following
let obj = {} let bad = 1 < obj.two let unexpected = 1 /> let nowFixed = 1 - The
<on line 4 is treated as a JSX tag opener (even when the self-closing tag on line 7 is removed)
Note how GitHub's highlighting in 2. and this screenshot disagree

- Declaring
twoslightly changes highlighting, but notice the.inobj.twois still green. Using "Developer: Inspect Editor Token and Scopes" shows it's still considered a JSX tag.

Extended example
To show all the specific conditions required to trigger this issue

Show code snippet for copy-paste
let expected, notExpected, obj = {}
let expectedFunc = () => { return (true) }
obj.twenty = 20
expected = 10 < obj.twenty
expected = 10
< 20
expected = 30
> obj.twenty
expected = 10
< obj.twenty;
expected = (10
< obj.twenty)
expected = 10 <
obj.twenty
notExpected = 10
< obj.twenty // <-- `<` at start of line misinterpreted as JSX
// syntax highlighting continues to be wonky
let notExpectedFunc = () => { return (true) }
/> // until a JSX closing tag is added
let expectedFunc = () => { return (true) }
Real-world example
In my repo, using prettier for formatting (specifically, a prettier fork that places operators at start of line instead of end) yields the following output, with the problem starting at line 77. Note how placing the cursor on the less-than op highlights the greater-than op on line 84, as if lines 77-84 are just one giant JSX opening tag declaration. Also note that this is a .js file with no mention nor import of React, and the prior examples also had language type explicitly set to JavaScript.

My preemptive apologies if this issue is a duplicate (I searched but found none) or if it belongs in a more specific repo.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the JavaScript example in VS Code with extensions disabled, then inspect the repository's TypeScript grammar files. The fix is done when a less-than operator at the start of a line is highlighted as an operator rather than opening JSX, and highlighting does not continue until a closing tag.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, typescript
- Domain
- tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100