Haskell tokenization incorrectly splits backslash operators and lambda expressions
- Dominant language
- Haskell
- Stars
- 218
- Forks
- 67
- PR merge metrics
- No merged PRs in 30d
Description
The Haskell syntax definition in Skylighting incorrectly tokenizes operators containing backslashes and lambda expressions, producing different results compared to GHC's own tokenizer.
## Issues
### 1. Lambda Expressions (`\x -> ...`)
Skylighting lumps the lambda backslash and variable together as a single token:
```haskell
\x -> x
```
- **GHC tokenizer:** `[λ, x, ->, x]` (converts `\` to Unicode lambda)
- **Skylighting:** `["\x ", "->", " x"]` (lumps `\x ` together)
### 2. Set Difference Operator (`\\`)
Entire expression lumped into one token:
```haskell
a \\ b
```
- **GHC tokenizer:** `[a, \\, b]`
- **Skylighting:** `["a \\\\ b"]`
### 3. Logical OR Operator (`\/`)
Splits inconsistently, attaching backslash to preceding text:
```haskell
a \/ b
```
- **GHC tokenizer:** `[a, \/, b]`
- **Skylighting:** `["a \\", "/", " b"]`
### 4. Logical AND Operator (`/\`)
Splits inconsistently, attaching backslash to following text:
```haskell
a /\ b
```
- **GHC tokenizer:** `[a, /\, b]`
- **Skylighting:** `["a ", "/", "\\ b"]`
### 5. Custom Backslash Operators (`\+`, `\>`, etc.)
Sometimes splits instead of treating as atomic operators:
```haskell
\+ 1
\> x
```
- **GHC tokenizer:** `[\+, 1]` and `[\>, x]`
- **Skylighting:** `["\\", "+", " ", 1]` and `["\\", ">", " x"]`
## Expected Behavior
The Haskell syntax definition should recognize:
1. Lambda expressions: `\` followed by an identifier should be separate tokens
2. Backslash operators: `\\`, `\/`, `/\`, and custom operators like `\+`, `\>` should be atomic operator tokens
3. These should match GHC's tokenization behavior
## Impact
This affects any tool using Skylighting for Haskell syntax highlighting/tokenization, including Pandoc, static site generators, documentation tools, and code formatters.
Contributor guide
No contributing guide indexed for this repository
Research direction
No file or test is named in the issue; start by locating Skylighting’s Haskell syntax definition and comparing its tokenization with the listed examples. Done means lambda expressions and backslash-containing operators are tokenized separately and atomically as described, matching the expected behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- haskell
- Domain
- tooling
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100