microsoft / microsoft/terminal

Half-width Katakana and (han)dakuten should not overlap/combine.

Open
#18,087 11 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Area-Fonts Area-Rendering Issue-Bug Product-Terminal
Dominant language
C++
Stars
105k
Forks
9.6k
Avg merge
3d 17h
Merged PRs (30d)
29

Description

### Windows Terminal version

1.23.2913.0

### Windows build number

10.0.26100.2033 ARM64

### Explanations

I believe there is an error in the code for grapheme clusters text width computation in the current version of Windows Terminal (tested in Preview and Canary).

Japanese in the terminal can be tricky. For historical reasons there are two sets of katakana, a full-width that fits square / double-cells like hiragana and kanji, and a half-width set that fits single cells like ASCII text does.
The problem is how these handle dakuten (and handakuten, but I'll use dakuten to refer to both from now on), which are the Japanese equivalent of accents, and like other diacritical marks, can be combining or not… We have 3 sets of them, a non-combining half-width version, a non-combining full-width version, and a combining full-width version, plus precomposed characters as well.

U+3099 ゛ COMBINING KATAKANA-HIRAGANA VOICED SOUND MARK
U+309A ゜ COMBINING KATAKANA-HIRAGANA SEMI-VOICED SOUND MARK
U+309B ゛ KATAKANA-HIRAGANA VOICED SOUND MARK
U+309C ゜ KATAKANA-HIRAGANA SEMI-VOICED SOUND MARK
U+FF9E ゙ HALFWIDTH KATAKANA VOICED SOUND MARK
U+FF9F ゚ HALFWIDTH KATAKANA SEMI-VOICED SOUND MARK

Take `Windows Terminal` written in Japanese: `ウィンドウズ・ターミナル`.
The `ド` is `ト` with an extra `゛` mark, and `ズ` is `ス` with an extra `゛` mark.
There are 46 katakana, plus 9 small forms, which required their own glyphs in old terminals and PCs, and a large part of them can combine with `゛` or/and `゜`, yielding an extra 30 common combined katakana, and some foreign sounds can be represented using less common combinations, for a total of 92 katakana glyphs variations. Add the Japanese punctuation characters, and we reach over 100 symbols.
So while glyphs representing the combined katakana+dakuten is desirable and better looking, old systems didn't combine them, and used the main katakana glyph, followed by the (han)dakuten glyph. This worked pretty well for half-width katakana, as they felt squeezed, and the dakuten as a second character cell basically just made those square again.

So in half-width katakana, Windows Terminal is written `ウィンドウズ・ターミナル`. Note how the `ド` is represented using the two glyphs `ド`, and `ズ` with the two glyphs `ズ`.

When handling them as grapheme clusters, is makes sense to handle the half-width katakana+dakuten as a single group, they should never be separated. But when displayed in a console or terminal, they are separate characters, and probably should be handled separately, as in legacy systems such as those using Shift-JIS (MS-DOS and Windows codepage 932), they were really separate characters and dakuten could be placed anywhere by themselves.
Even more important, when displaying them, they do not combine or overlap!

The following behavior is the correct and expected way to show them in a terminal:
![Image](https://github.com/user-attachments/assets/247ae0e1-03db-41c0-abdb-8bdcea52c0b7)

And is the way it works in Windows Terminal Canary with the `wcswidth` text measurement mode.

But when using the `Grapheme clusters` text measurement mode, half-width handakuten are handled like combining diacritic, overlapping the previous katakana:
![Image](https://github.com/user-attachments/assets/d5b38661-6757-4b9c-9486-3bc4b463a806)

So to be clear, `U+3099` and `U+309A` are full-width combining, while `U+309B` and `U+309C` are full-width, `U+FF9E` and `U+FF9F` are half-width, all non-combining.

`ウィンドウズ・ターミナル` is full-width using precomposed characters, `ウィンドウズ・ターミナル` is full-width using combining dakuten, `ウィント゛ウス゛・ターミナル` is full-width using non-combining dakuten, and `ウィンドウズ・ターミナル` is half-width, which is always non-combining.

I think for Windows Terminal, the grapheme clusters code should not group half-width katakana with dakuten or handakuten. It would fix the text measurement issue and users probably expect to be able to navigate between those characters as if they were completely separate for cursor navigation.

### Expected Behavior

Half-width katakana shouldn't have dakuten overlapping them.

### Actual Behavior

Half-width katakana has dakuten overlapping them as if they were combining diacritical marks.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the grapheme clusters text measurement mode and compare its handling of half-width Katakana with the wcswidth mode. Reproduce the issue using the half-width examples ウィンドウズ・ターミナル and the U+FF9E/U+FF9F marks. Done means half-width dakuten and handakuten remain separate, non-overlapping cells and cursor navigation treats them separately.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
cli, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.