Feature request: provide Template Haskell quasiquoter that generates lexer definition in-place
- Dominant language
- Haskell
- Stars
- 323
- Forks
- 87
- PR merge metrics
- No merged PRs in 30d
Description
Currently `alex` is a command-line tool that takes files in its own format and relies on support from Cabal-the-library to invoke `alex` command in order to preprocess `.x` files before passing them to ghc. This works reasonably well, but I think there's something to be gained if `alex` package would provide quasiquoter that takes lexer definition in the currently used format but spits out TH expressions instead of generating new file.
E.g. take `tokens_scan_user.x` test from the alex test suite. Instead of
```haskell
{
module Main (main) where
import System.Exit
}
%wrapper "basic" -- Defines: AlexInput, alexGetByte, alexPrevChar
$digit = 0-9
$alpha = [a-zA-Z]
$ws = [\ \t\n]
tokens :-
5 / {\ u _ibt _l _iat -> u == FiveIsMagic} { \s -> TFive (head s) }
$digit { \s -> TDigit (head s) }
$alpha { \s -> TAlpha (head s) }
$ws { \s -> TWSpace (head s) }
{
data Token = TDigit Char
| TAlpha Char
| TWSpace Char
| TFive Char -- Predicated only
| TLexError
deriving (Eq,Show)
data UserLexerMode = NormalMode
| FiveIsMagic
deriving Eq
main | test1 /= result1 = exitFailure
| test2 /= result2 = exitFailure
-- all succeeded
| otherwise = exitWith ExitSuccess
run_lexer :: UserLexerMode -> String -> [Token]
run_lexer m s = go ('\n', [], s)
where go i@(_,_,s') = case alexScanUser m i 0 of
AlexEOF -> []
AlexError _i -> [TLexError]
AlexSkip i' _len -> go i'
AlexToken i' len t -> t (take len s') : go i'
test1 = run_lexer FiveIsMagic "5 x"
result1 = [TFive '5',TWSpace ' ',TAlpha 'x']
test2 = run_lexer NormalMode "5 x"
result2 = [TDigit '5',TWSpace ' ',TAlpha 'x']
}
```
I'd like to write `TokensScanUser.hs` file that looks like:
```haskell
{-# LANGUAGE QuasiQuotes #-}
module Main (main) where
import System.Exit
import Alex.TH
genLexer defaultLexer [alex|
%wrapper "basic" -- Defines: AlexInput, alexGetByte, alexPrevChar
$digit = 0-9
$alpha = [a-zA-Z]
$ws = [\ \t\n]
tokens :-
5 / {\u _ibt _l _iat -> u == FiveIsMagic} { \s -> TFive (head s) }
$digit { \s -> TDigit (head s) }
$alpha { \s -> TAlpha (head s) }
$ws { \s -> TWSpace (head s) }
|]
data Token = TDigit Char
| TAlpha Char
| TWSpace Char
| TFive Char -- Predicated only
| TLexError
deriving (Eq,Show)
data UserLexerMode = NormalMode
| FiveIsMagic
deriving Eq
main | test1 /= result1 = exitFailure
| test2 /= result2 = exitFailure
-- all succeeded
| otherwise = exitWith ExitSuccess
run_lexer :: UserLexerMode -> String -> [Token]
run_lexer m s = go ('\n', [], s)
where go i@(_,_,s') = case alexScanUser m i 0 of
AlexEOF -> []
AlexError _i -> [TLexError]
AlexSkip i' _len -> go i'
AlexToken i' len t -> t (take len s') : go i'
test1 = run_lexer FiveIsMagic "5 x"
result1 = [TFive '5',TWSpace ' ',TAlpha 'x']
test2 = run_lexer NormalMode "5 x"
result2 = [TDigit '5',TWSpace ' ',TAlpha 'x']
```
Having such quasiquoter will provide following benefits:
- This will provide an option to use alex independent of system's preprocessor if, say, `clang` starts to behave funny
- This can help end confusion of text editors with overly long lines as mentioned #84
- There will be no nowelines in the generated files (becase there will be no generated files) that may annoy someone #105
- User will be able to just start `ghci` in his or hers project and load lexer definition to play with - no need to add `dist/build/` directory (or different directory, depending on build target and the build tool (`cabal` has one prefix here, `stack` has another depending on the snapshot))to `ghci` path any more
- Indexing with e.g. tag generators should also improve because these programs will just skip quasiquoter part and index all user-defined functions.
Contributor guide
Research direction
Start with the alex test-suite case tokens_scan_user.x and the existing command-line preprocessing path, then compare it with the proposed TokensScanUser.hs example and Alex.TH import. Done means a quasiquoter accepts the existing lexer-definition format and produces equivalent Template Haskell expressions without a generated file; the payload does not name a specific implementation file or test command.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- haskell
- Domain
- compilers, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100