notepad-plus-plus / notepad-plus-plus/notepad-plus-plus
Regex error dealing with characters outside BMP
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 29.4k
- Forks
- 5.4k
- PR merge metrics
- No merged PRs in 30d
Description
Description of the Issue
When using the built-in regex to do a simple character replacement with CJK characters, I found a bug in the replaced results. Characters that are in the Basic Multilingual Plane (BMP) of Unicode are replaced as-is (CJK Unified Ideograph and Extension-A), but characters out of the BMP (CJK Unified Ideograph Extension-B to -G) are replaced wrongly.
Steps to Reproduce the Issue
-
Copy or download the following text:
𠊎𠎷𠕆𠖄𠗻𠞩𠞭𠠃𠠝𠢕𠯗𠲿𡳞𢎙𢪱𢯭𢯾𢱤𢲴𢲸𢳪𢶀𢼌𢼛𣁳𣮈𣲩𣼎𤊶𤍒𤐙𤐰𤘅𤞚𤲍𤶃𤸁𤸱𤺅𤺪𥌚𥍉𥐵𥯟𥯥𥰔𥴊𥺆𦉎𦊓𦜆𦟪𧉟𧌄𧩣𧮙𧻴𧿬𧿳𨂿𨃰𨅔𨒇𨢑𩑾𩚨𩛩𩜄𩜰𩵱𩸙𩸶𩼧𪁎𪐞𪖐𪘒𪜶𪹚𫝏𫝘𫝛𫝺𫝻𫞼𫟂𫟧𫠛㐁㑁㓾㔂㕸㖅㖸㗘㗱㘔㘛㘝㜮㜷㝗㝩㤉㤕㧁㧌㧎㧒㧡㧣㧻㧾㨂㨑㨨㨪㨻㩆㪐㬹㰵㴘㴙㷮㸐㸰㼓㽎㾀㾊㿸䀐䀯䀴䁯䃗䆀䇶䈄䉂䌈䖙䖳䗁䗆䗐䘆䘥䘼䞚䟓䟘䟩䠡䢍䢢䫌䯋䲅亼伨倈倲傪僫凃凴刣剾勥匧叹吂吓吔哊哯哹唩唿啉啲喺嗬噍噼嚟圝坔塠壠奒媠嫲廍抅抐抺拁挱挵挷捒捝捾揞揢揬揻搙搝搩摖摱撆撍撨攋攑敋敱杮桊梘槺橂檫溚漖灇炣烌煏熻燶疴癀眡砄砛硓礮秥稴竳竵篼粩糍絚繏羓羴肨胮胿腈膁膎膭臁臽蕒蕳藠虬蝲螕蟎蟮蠘襀襇誐贌跔踭蹽躼軁递逩鈪錑閕靗頕魩鮊鮔鮘鯮鱲黗𠕇𠜱𡢃𢫦𢳆𣛮𣻸𤉙𤌍𥉌𥑮𧊅𨂾𫣆𬠖𬦰𰣻𰹬
huninn-1.1-new.txt -
Do a regex searching for (.) and replacing with $1\r\n
Expected Behavior
Each character are supposed to be changed to 1 character per line. Attached image is done with manual editing.

Actual Behavior
Characters outside BMP are replaced with wrong character. Suspiciously it maintain the last 4 digit of the Unicode character with xF0 in front.

After 88th character (after CJK Ext-B to -F, it returned to BMP):

After 296th character (CJK Ext-B to -G)

(index are wrong due to new line at 1st line)
Debug Information
Notepad++ v7.8.5 (32-bit)
Build time : Mar 4 2020 - 11:04:20
Path : C:\Program Files (x86)\Notepad++\notepad++.exe
Admin mode : OFF
Local Conf mode : OFF
OS Name : Windows 10 Enterprise (64-bit)
OS Version : 1909
OS Build : 18363.778
Plugins : JSMinNPP.dll LuaScript.dll mimeTools.dll NppConverter.dll NppExport.dll Tidy2.dll XBrackets.dll XMLTools.dll
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No source file or test entry point is named. Reproduce the built-in regex replacement with the supplied Unicode text and (.) replaced by $1\r\n; done means every BMP and non-BMP character remains intact and appears on its own line.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- desktop
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100