UTF-8 and text 2.0
- Dominant language
- Haskell
- Stars
- 323
- Forks
- 87
- PR merge metrics
- No merged PRs in 30d
Description
As the documentation says, Alex works over a stream of UTF-8 encoded bytes, retrieved one at a time by `alexGetByte`.
> Lexer specifications are written in terms of Unicode characters, but Alex works internally on a UTF-8 encoded byte sequence.
>
> Depending on how you use Alex, the fact that Alex uses UTF-8 encoding internally may or may not affect you. If you use one of the wrappers (below) that takes input from a Haskell String, then the UTF-8 encoding is handled automatically. However, if you take input from a ByteString, then it is your responsibility to ensure that the input is properly UTF-8 encoded.
From an external viewpoint as a consumer (I am not familiar with how Alex is implemented), this seems like a strange design decision to me. If my source is already in a UTF-8 format like `String` or `Data.Text(.Lazy).Text` (with the new text 2.0 release), it seems that in order to run Alex on it, I (or a wrapper) would have to write logic to decode the UTF-8 content into individual bytes, just so that Alex can immediately re-encode it back into UTF-8 internally.
So I'm wondering if there's a reason that Alex needs to work over bytes and not `Char`s, or if that was perhaps done to support lexing `ByteString`s directly, without having to first unpack the data into UTF-8 `String`s or decode into UTF-16 `Text`s (with text <2.0), which would be unnecessary overhead either way.
If Alex has to work over bytes for internal reasons, I think it would be a good idea to implement new wrappers for the UTF-8 `Text` types, since I'd imagine that would be a pretty common use case.
Otherwise, would it be possible to expose an `alexGetChar`-based interface that simply skips the UTF-8 decoding portion of Alex's internal logic, which would be more ergonomic and efficient for UTF-8 based types?
Contributor guide
Assessment
This issue has not been assessed yet.