haskell / haskell/alex

basic-bytestring wrapper does not work with left contexts

Open
#53 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Haskell
Stars
323
Forks
87
PR merge metrics
No merged PRs in 30d

Description

The `basic-bytestring` wrapper does not work correctly with left contexts when provided with characters which are encoded as multiple bytes in UTF-8.

The following program produces `True,False` while I expect it to produce `True,True`.

```
{
{-# LANGUAGE OverloadedStrings #-}
module Main where
import qualified Data.ByteString.Lazy.Char8 as B
}

%wrapper "basic-bytestring"

tokens :-
a^b {const True}
a {const True}
∃^∀ {const True}
∃ {const True}
. {const False}

{
main::IO ()
main = do
print . and . alexScanTokens $ "ab"
print . and . alexScanTokens $ "∃∀"
}
```

I think this is due to [`alexGetByte`](https://github.com/simonmar/alex/blob/master/templates/wrappers.hs#L104) for this wrapper remembering the last byte rather than the last _character_.

Since converting input bytes to characters puts unnecessary costs on the users of this wrapper maybe we should just not implement left contexts in this case?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.