google / google/emoji-segmenter

Classification issues

Open
#17 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
70
Forks
19
PR merge metrics
No merged PRs in 30d

Description

While working on an alternative emoji segmentation implementation I noticed three presentation classification issues in the Ragel grammar:

**Bare `Emoji_Modifier` misclassified as text**
A lone modifier has `Emoji_Presentation=Yes` and should be classified as `is_emoji=true`. However, since `EMOJI_MODIFIER` is not included in `any_emoji`, it falls through to `text_run` and is classified as `is_emoji=false`.

**Bare `Emoji_Modifier_Base` misclassified as emoji**
[Nine codepoints](https://util.unicode.org/UnicodeJsps/list-unicodeset.jsp?a=%5B%3AEmoji_Modifier_Base%3A%5D%26%5B%3AEmoji_Presentation%3DNo%3A%5D) in the `EMOJI_MODIFIER_BASE` category have `Emoji_Presentation=No`, because `EMOJI_MODIFIER_BASE` is listed directly in `emoji_presentation`, a bare occurrence is classified as `is_emoji=true` regardless of the codepoint's default presentation property.

**Bare `Regional_Indicator` misclassified as text**
A lone regional indicator has `Emoji_Presentation=Yes` and should be classified as `is_emoji=true`. However, since `emoji_presentation` only includes `emoji_flag_sequence` (a pair of regional indicators), a single regional indicator falls through to `text_run` and is incorrectly classified as `is_emoji=false`.

**Unqualified keycap sequence not recognised**
An unqualified keycap sequence (`KEYCAP_BASE` + `COMBINING_ENCLOSING_KEYCAP`, e.g. `U+0023 U+20E3`) is not matched by any emoji rule. `emoji_keycap_sequence` requires an intervening VS-15 or VS-16, so the two codepoints are scanned separately as individual `text_run` tokens. Unicode includes unqualified keycap sequences in `emoji-test.txt` and they appear in legacy content, so failing to recognise them as a single emoji sequence is likely incorrect.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.