mb_strtoupper converts non-alphabetic characters in Shift_JIS (CP932)
Nobody has claimed this yet.
- Dominant language
- C
- Stars
- 40.4k
- Forks
- 8.1k
- Avg merge
- 2d 13h
- Merged PRs (30d)
- 96
Description
Description
mb_strtoupper converts non-alphabetic characters in Shift_JIS (CP932)
var_dump(
"\x81\xE0" === mb_strtoupper("\x87\x90", 'cp932')
);
\x81\xE0 and \x87\x90 mean ≒ (U+2252: Approximately Equal to or the Image Of).
var_dump(
"\u{2252}" === mb_convert_encoding("\x81\xE0", 'utf-8', 'cp932'),
"\u{2252}" === mb_convert_encoding("\x87\x90", 'utf-8', 'cp932')
);
\x87\x90 is the one of NEC special characters and was registered in duplicate for historical reasons.
The unintended conversion is caused by a round-trip conversion between Unicode and Shift_JIS.
var_dump(
"\x81\xE0" === roundtrip("\x87\x90", 'cp932'),
"\x81\xE0" === roundtrip("\x81\xE0", 'cp932')
);
function roundtrip($char, $enc) {
return mb_convert_encoding(mb_convert_encoding($char, 'utf-8', $enc), $enc, 'utf-8');
}
As far as I know, 398 characters in Shift_JIS are affected by round-trip conversion. The test code is here.
The same problem applies to mb_scrub, mb_strtolower, mb_convert_case.
var_dump(
"\x81\xE0" === mb_scrub("\x87\x90", 'cp932'),
"\x81\xE0" === mb_strtolower("\x87\x90", 'cp932'),
"\x81\xE0" === mb_strtoupper("\x87\x90", 'cp932'),
"\x81\xE0" === mb_convert_case("\x87\x90", MB_CASE_TITLE, 'cp932')
);
PHP Version
PHP 8.2.10
Operating System
Debian 11.7 (Google ChromeOS 117.0)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the supplied CP932 examples and use the mb_scrub, mb_strtolower, mb_strtoupper, and mb_convert_case entry points as the starting scope. Done means the duplicate NEC character sequence remains distinct from its canonical CP932 sequence instead of being changed by round-trip conversion, with the affected behavior covered by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- php
- Domain
- internationalization
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100