notepad-plus-plus / notepad-plus-plus/notepad-plus-plus

Regex error dealing with characters outside BMP

Open
#8,173 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

boost regex dependent
Dominant language
C++
Stars
29.4k
Forks
5.4k
PR merge metrics
No merged PRs in 30d

Description

Description of the Issue

When using the built-in regex to do a simple character replacement with CJK characters, I found a bug in the replaced results. Characters that are in the Basic Multilingual Plane (BMP) of Unicode are replaced as-is (CJK Unified Ideograph and Extension-A), but characters out of the BMP (CJK Unified Ideograph Extension-B to -G) are replaced wrongly.

Steps to Reproduce the Issue
  1. Copy or download the following text:
    𠊎𠎷𠕆𠖄𠗻𠞩𠞭𠠃𠠝𠢕𠯗𠲿𡳞𢎙𢪱𢯭𢯾𢱤𢲴𢲸𢳪𢶀𢼌𢼛𣁳𣮈𣲩𣼎𤊶𤍒𤐙𤐰𤘅𤞚𤲍𤶃𤸁𤸱𤺅𤺪𥌚𥍉𥐵𥯟𥯥𥰔𥴊𥺆𦉎𦊓𦜆𦟪𧉟𧌄𧩣𧮙𧻴𧿬𧿳𨂿𨃰𨅔𨒇𨢑𩑾𩚨𩛩𩜄𩜰𩵱𩸙𩸶𩼧𪁎𪐞𪖐𪘒𪜶𪹚𫝏𫝘𫝛𫝺𫝻𫞼𫟂𫟧𫠛㐁㑁㓾㔂㕸㖅㖸㗘㗱㘔㘛㘝㜮㜷㝗㝩㤉㤕㧁㧌㧎㧒㧡㧣㧻㧾㨂㨑㨨㨪㨻㩆㪐㬹㰵㴘㴙㷮㸐㸰㼓㽎㾀㾊㿸䀐䀯䀴䁯䃗䆀䇶䈄䉂䌈䖙䖳䗁䗆䗐䘆䘥䘼䞚䟓䟘䟩䠡䢍䢢䫌䯋䲅亼伨倈倲傪僫凃凴刣剾勥匧叹吂吓吔哊哯哹唩唿啉啲喺嗬噍噼嚟圝坔塠壠奒媠嫲廍抅抐抺拁挱挵挷捒捝捾揞揢揬揻搙搝搩摖摱撆撍撨攋攑敋敱杮桊梘槺橂檫溚漖灇炣烌煏熻燶疴癀眡砄砛硓礮秥稴竳竵篼粩糍絚繏羓羴肨胮胿腈膁膎膭臁臽蕒蕳藠虬蝲螕蟎蟮蠘襀襇誐贌跔踭蹽躼軁递逩鈪錑閕靗頕魩鮊鮔鮘鯮鱲黗𠕇𠜱𡢃𢫦𢳆𣛮𣻸𤉙𤌍𥉌𥑮𧊅𨂾𫣆𬠖𬦰𰣻𰹬
    huninn-1.1-new.txt

  2. Do a regex searching for (.) and replacing with $1\r\n

Expected Behavior

Each character are supposed to be changed to 1 character per line. Attached image is done with manual editing.
image

Actual Behavior

Characters outside BMP are replaced with wrong character. Suspiciously it maintain the last 4 digit of the Unicode character with xF0 in front.
image
After 88th character (after CJK Ext-B to -F, it returned to BMP):
image
After 296th character (CJK Ext-B to -G)
image
(index are wrong due to new line at 1st line)

Debug Information

Notepad++ v7.8.5 (32-bit)
Build time : Mar 4 2020 - 11:04:20
Path : C:\Program Files (x86)\Notepad++\notepad++.exe
Admin mode : OFF
Local Conf mode : OFF
OS Name : Windows 10 Enterprise (64-bit)
OS Version : 1909
OS Build : 18363.778
Plugins : JSMinNPP.dll LuaScript.dll mimeTools.dll NppConverter.dll NppExport.dll Tidy2.dll XBrackets.dll XMLTools.dll

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No source file or test entry point is named. Reproduce the built-in regex replacement with the supplied Unicode text and (.) replaced by $1\r\n; done means every BMP and non-BMP character remains intact and appears on its own line.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
desktop
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.