cdepillabout / cdepillabout/highlight

support utf8 regexes

Open
#5 1 comment 0 reactions 0 assignees View on GitHub
beginner-friendly help wanted
Dominant language
Haskell
Stars
4
Forks
1
PR merge metrics
No merged PRs in 30d

Description

It would be nice to support UTF8 regexes.

Here is an example of doing an UTF8 regex with grep:

![utf-grep-example](https://user-images.githubusercontent.com/64804/27727002-acf63e84-5db7-11e7-916b-651f4e74c07d.png)

Here is what happens when using highlight:

![highlight-bad-utf8](https://user-images.githubusercontent.com/64804/27727217-765fcdee-5db8-11e7-8cfe-4ebb3ea82c0a.png)

Note that highlight is just working on a character-by-character basis, so it is possible to do a regex on japanese if you account for most japanese characters being 3 bytes in utf8:

![highlight-good-utf8](https://user-images.githubusercontent.com/64804/27727265-b4332ad0-5db8-11e7-87e0-04c55f610c50.png)

When running the three previous examples, my locale settings are as follows:

```sh
$ locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=
```

Changing it to `LC_ALL=en_US.ASCII` makes `grep` ignore utf8 and output the same thing as `highlight`.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.