jgm / jgm/skylighting

Haskell tokenization incorrectly splits backslash operators and lambda expressions

Open
#209 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Haskell
Stars
218
Forks
67
PR merge metrics
No merged PRs in 30d

Description

The Haskell syntax definition in Skylighting incorrectly tokenizes operators containing backslashes and lambda expressions, producing different results compared to GHC's own tokenizer.

## Issues

### 1. Lambda Expressions (`\x -> ...`)
Skylighting lumps the lambda backslash and variable together as a single token:
```haskell
\x -> x
```
- **GHC tokenizer:** `[λ, x, ->, x]` (converts `\` to Unicode lambda)
- **Skylighting:** `["\x ", "->", " x"]` (lumps `\x ` together)

### 2. Set Difference Operator (`\\`)
Entire expression lumped into one token:
```haskell
a \\ b
```
- **GHC tokenizer:** `[a, \\, b]`
- **Skylighting:** `["a \\\\ b"]`

### 3. Logical OR Operator (`\/`)
Splits inconsistently, attaching backslash to preceding text:
```haskell
a \/ b
```
- **GHC tokenizer:** `[a, \/, b]`
- **Skylighting:** `["a \\", "/", " b"]`

### 4. Logical AND Operator (`/\`)
Splits inconsistently, attaching backslash to following text:
```haskell
a /\ b
```
- **GHC tokenizer:** `[a, /\, b]`
- **Skylighting:** `["a ", "/", "\\ b"]`

### 5. Custom Backslash Operators (`\+`, `\>`, etc.)
Sometimes splits instead of treating as atomic operators:
```haskell
\+ 1
\> x
```
- **GHC tokenizer:** `[\+, 1]` and `[\>, x]`
- **Skylighting:** `["\\", "+", " ", 1]` and `["\\", ">", " x"]`

## Expected Behavior

The Haskell syntax definition should recognize:
1. Lambda expressions: `\` followed by an identifier should be separate tokens
2. Backslash operators: `\\`, `\/`, `/\`, and custom operators like `\+`, `\>` should be atomic operator tokens
3. These should match GHC's tokenization behavior

## Impact

This affects any tool using Skylighting for Haskell syntax highlighting/tokenization, including Pandoc, static site generators, documentation tools, and code formatters.

Contributor guide

No contributing guide indexed for this repository

Research direction

No file or test is named in the issue; start by locating Skylighting’s Haskell syntax definition and comparing its tokenization with the listed examples. Done means lambda expressions and backslash-containing operators are tokenized separately and atomically as described, matching the expected behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
haskell
Domain
tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.