haskell / haskell/alex

Feature request: provide Template Haskell quasiquoter that generates lexer definition in-place

Open
#108 1 comment 3 reactions 0 assignees View on GitHub
Dominant language
Haskell
Stars
323
Forks
87
PR merge metrics
No merged PRs in 30d

Description

Currently `alex` is a command-line tool that takes files in its own format and relies on support from Cabal-the-library to invoke `alex` command in order to preprocess `.x` files before passing them to ghc. This works reasonably well, but I think there's something to be gained if `alex` package would provide quasiquoter that takes lexer definition in the currently used format but spits out TH expressions instead of generating new file.

E.g. take `tokens_scan_user.x` test from the alex test suite. Instead of

```haskell
{
module Main (main) where
import System.Exit
}

%wrapper "basic" -- Defines: AlexInput, alexGetByte, alexPrevChar

$digit = 0-9
$alpha = [a-zA-Z]
$ws = [\ \t\n]

tokens :-

5 / {\ u _ibt _l _iat -> u == FiveIsMagic} { \s -> TFive (head s) }
$digit { \s -> TDigit (head s) }
$alpha { \s -> TAlpha (head s) }
$ws { \s -> TWSpace (head s) }

{

data Token = TDigit Char
| TAlpha Char
| TWSpace Char
| TFive Char -- Predicated only
| TLexError
deriving (Eq,Show)

data UserLexerMode = NormalMode
| FiveIsMagic
deriving Eq

main | test1 /= result1 = exitFailure
| test2 /= result2 = exitFailure
-- all succeeded
| otherwise = exitWith ExitSuccess

run_lexer :: UserLexerMode -> String -> [Token]
run_lexer m s = go ('\n', [], s)
where go i@(_,_,s') = case alexScanUser m i 0 of
AlexEOF -> []
AlexError _i -> [TLexError]
AlexSkip i' _len -> go i'
AlexToken i' len t -> t (take len s') : go i'

test1 = run_lexer FiveIsMagic "5 x"
result1 = [TFive '5',TWSpace ' ',TAlpha 'x']

test2 = run_lexer NormalMode "5 x"
result2 = [TDigit '5',TWSpace ' ',TAlpha 'x']
}
```

I'd like to write `TokensScanUser.hs` file that looks like:
```haskell
{-# LANGUAGE QuasiQuotes #-}

module Main (main) where
import System.Exit

import Alex.TH

genLexer defaultLexer [alex|
%wrapper "basic" -- Defines: AlexInput, alexGetByte, alexPrevChar

$digit = 0-9
$alpha = [a-zA-Z]
$ws = [\ \t\n]

tokens :-

5 / {\u _ibt _l _iat -> u == FiveIsMagic} { \s -> TFive (head s) }
$digit { \s -> TDigit (head s) }
$alpha { \s -> TAlpha (head s) }
$ws { \s -> TWSpace (head s) }
|]

data Token = TDigit Char
| TAlpha Char
| TWSpace Char
| TFive Char -- Predicated only
| TLexError
deriving (Eq,Show)

data UserLexerMode = NormalMode
| FiveIsMagic
deriving Eq

main | test1 /= result1 = exitFailure
| test2 /= result2 = exitFailure
-- all succeeded
| otherwise = exitWith ExitSuccess

run_lexer :: UserLexerMode -> String -> [Token]
run_lexer m s = go ('\n', [], s)
where go i@(_,_,s') = case alexScanUser m i 0 of
AlexEOF -> []
AlexError _i -> [TLexError]
AlexSkip i' _len -> go i'
AlexToken i' len t -> t (take len s') : go i'

test1 = run_lexer FiveIsMagic "5 x"
result1 = [TFive '5',TWSpace ' ',TAlpha 'x']

test2 = run_lexer NormalMode "5 x"
result2 = [TDigit '5',TWSpace ' ',TAlpha 'x']
```

Having such quasiquoter will provide following benefits:
- This will provide an option to use alex independent of system's preprocessor if, say, `clang` starts to behave funny
- This can help end confusion of text editors with overly long lines as mentioned #84
- There will be no nowelines in the generated files (becase there will be no generated files) that may annoy someone #105
- User will be able to just start `ghci` in his or hers project and load lexer definition to play with - no need to add `dist/build/` directory (or different directory, depending on build target and the build tool (`cabal` has one prefix here, `stack` has another depending on the snapshot))to `ghci` path any more
- Indexing with e.g. tag generators should also improve because these programs will just skip quasiquoter part and index all user-defined functions.

Contributor guide

Open the contributing guide

Research direction

Start with the alex test-suite case tokens_scan_user.x and the existing command-line preprocessing path, then compare it with the proposed TokensScanUser.hs example and Alex.TH import. Done means a quasiquoter accepts the existing lexer-definition format and produces equivalent Template Haskell expressions without a generated file; the payload does not name a specific implementation file or test command.

Written by the indexing model from the issue text.

Assessment

Tech stack
haskell
Domain
compilers, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.