haskell / haskell/text

Decoupling byte-level encoding

未关闭
#535 20 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Haskell
星标
421
派生
163
PR 合并指标
30 天内没有已合并 PR

描述

When writing a JSON parser (GaloisInc/json#17) I needed some way to decode UTF-8 and to my dismay I found all existing solutions do not fit my expectations:
- `GHC.Encoding.UTF8` and `GHC.IO.Encoding` are `IO`-based and I don't want that in a parser;
- `Data.Text.Internal.Encoding.Utf8`, while pure, appears to both return `Reject` as an error and has a rather complex interface;
- `Data.Text.Encoding.*` and `Data.Text.Lazy.Encoding.*` are already parsers themselves, too high-level for this task;
- `utf-string`'s `Codec.Binary.UTF8.String` consumes and returns lists, so it isn't parser-compatible.

I decided to handroll the UTF-8 decoding, which allowed me to categorize the errors (see [Encoding.Mixed.Error](https://github.com/BurningWitness/json/blob/ea8d0c80/src/Encoding/Mixed/Error.hs#L7-L51)) and resulted in a lot of code on the parser side that has little to do with consuming bytes per se (see [Codec.Web.JSON.Parse.String](https://github.com/BurningWitness/json/blob/ea8d0c80/src/Codec/Web/JSON/Parse/String.hs#L64-L286)).

However the code I wrote can instead be generalized to:

```haskell
-- Assume Error is Encoding.Mixed.Error.Error

data UTF8 a = UTF8_1 a
| Part_2 (Word8 -> UTF8_2 a)
| Part_3_1 (Word8 -> Part_3_1 a)
| Part_4_1 (Word8 -> Part_4_1 a)
| Error_1 Error

data UTF8_2 a = UTF8_2 a
| Error_2 Error

data Part_3_1 a = Part_3_2 (Word8 -> UTF8_3 a)
| Error_3_1 Error

data UTF8_3 a = UTF8_3 a
| Error_3_2 Error

data Part_4_1 a = Part_4_2 (Word8 -> Part_4_2 a)
| Error_4_1 Error

data Part_4_2 a = Part_4_3 (Word8 -> UTF8_4 a)
| Error_4_2 Error

data UTF8_4 a = UTF8_4 a
| Error_4_3 Error

newtype Conv1 a = Conv1 (Word8 -> a)
newtype Conv2 a = Conv2 (Word8 -> Word8 -> a)
newtype Conv3 a = Conv3 (Word8 -> Word8 -> Word8 -> a)
newtype Conv4 a = Conv4 (Word8 -> Word8 -> Word8 -> Word8 -> a)

utf8 :: Conv1 a -> Conv2 a -> Conv3 a -> Conv4 a -> Word8 -> UTF8 a
utf8 = -- I'm omitting the implementation, but it's only 50 lines long
```

Parsing then is simply unwrapping `UTF8`. This decouples character validation and conversion, the only part of decoding left is ensuring only the maximal subpart of an ill-formed sequence is consumed, which is the responsibility of the parser.

---

My proposal is creating a separate package with a focus specifically on decoding/encoding UTF-8/UTF-16/UTF-32 on byte-level. Then `text` can drop some internal modules in favor of a simpler common interface.

This proposal is however naive: I do not know whether GHC can inline these datatypes reliably or, indeed, at all. Based on my cursory reading of the [Secrets of the Glasgow Haskell Compiler inliner](https://www.microsoft.com/en-us/research/wp-content/uploads/2002/07/inline.pdf) paper it should, as each of these expressions is *trivial*.

This doesn't clash with the issue of *GHC's many UTF-8 implementations* (outlined in [`GHC.Encoding.UTF8`](https://hackage.haskell.org/package/base-4.18.0.0/docs/src/GHC.Encoding.UTF8.html)) as all other algorithms are in `IO`.

Other concerns:
- `text` is a core library, so I assume an extra dependency can't just be added on a whim;
- Package named [`utf`](https://hackage.haskell.org/package/utf) already exists and is deprecated. I don't know how hard reclaiming deprecated packages is.

贡献指南

这个仓库没有索引到贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。