haskell / haskell/bytestring

Surprising behavior of ByteString literals via IsString

Open
#140 151 comments 1 reaction 0 assignees View on GitHub
blocked: ghc documentation pitfall
Dominant language
Haskell
Stars
301
Forks
144
Avg merge
7d 22h
Merged PRs (30d)
1

Description

At work, we discovered a somewhat surprising behavior of ByteString's `IsString` instance and interaction with `OverloadedStrings`.

The following REPL session demonstrates the issue:

```
λ> BS.unpack $ T.encodeUtf8 ("bla語" :: Text)
[98,108,97,232,170,158]
λ> BS.unpack $ ("bla語" :: BS.ByteString)
[98,108,97,158]
λ> T.decodeUtf8 $ ("bla語" :: BS.ByteString)
*** Exception: Cannot decode byte '\x9e': Data.Text.Internal.Encoding.decodeUtf8: Invalid UTF-8
```

The `IsString` instance calls `packChars` which calls [`c2w`](https://github.com/haskell/bytestring/blob/master/Data/ByteString/Internal.hs#L529-L534), which silently truncates the bytes.

I'd be happy to put together a PR to document the behavior of the `IsString` instance.

I think I expected it to encode the string using the source encoding. I don't know whether or not that's a feasible or desirable change.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.