basic-bytestring wrapper does not work with left contexts
- Dominant language
- Haskell
- Stars
- 323
- Forks
- 87
- PR merge metrics
- No merged PRs in 30d
Description
The `basic-bytestring` wrapper does not work correctly with left contexts when provided with characters which are encoded as multiple bytes in UTF-8.
The following program produces `True,False` while I expect it to produce `True,True`.
```
{
{-# LANGUAGE OverloadedStrings #-}
module Main where
import qualified Data.ByteString.Lazy.Char8 as B
}
%wrapper "basic-bytestring"
tokens :-
a^b {const True}
a {const True}
∃^∀ {const True}
∃ {const True}
. {const False}
{
main::IO ()
main = do
print . and . alexScanTokens $ "ab"
print . and . alexScanTokens $ "∃∀"
}
```
I think this is due to [`alexGetByte`](https://github.com/simonmar/alex/blob/master/templates/wrappers.hs#L104) for this wrapper remembering the last byte rather than the last _character_.
Since converting input bytes to characters puts unnecessary costs on the users of this wrapper maybe we should just not implement left contexts in this case?
Contributor guide
Assessment
This issue has not been assessed yet.