microsoft / microsoft/Power-Fx
Incorrect results for string functions on args with higher-plane characters
Nobody has claimed this yet.
- Dominant language
- C#
- Stars
- 3.4k
- Forks
- 358
- Avg merge
- 10h 34m
- Merged PRs (30d)
- 3
Description
Some functions are improperly treating strings that contain higher-plane Unicode characters (i.e., those with code points at U+10000 and higher, including most emoji) as if each of those characters were two characters long and unintelligible.
Split("🦓🦊🐺𐊀","")
→
Table({Value:"�"},{Value:"�"},{Value:"�"},{Value:"�"},{Value:"�"},{Value:"�"},{Value:"�"},{Value:"�"})
(Should be Table({Value:"🦓"},{Value:"🦊"},{Value:"🐺"},{Value:"𐊀"}))
Left("xyz🅰🅱🅲", 6)
→
"xyz🅰�" (note the ending character which is apparently half of the 🅱 emoji; this should just evaluate to the same string as was passed in)
Similarly, Right("🅰🅱🅲def", 6) → "�🅲def", and Mid("🅰🅱🅲def", 2, 4) → "�🅱�".
I believe something is going awry with how PowerFx is handling characters wider than 16 bits, and strings aren't being kept in a consistent translation format, which is leading to these errors.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the Split, Left, Right, and Mid examples from the issue with higher-plane Unicode characters. Trace the string handling used by these functions and add coverage for the shown cases; done means each function preserves complete Unicode characters without replacement characters or split surrogate pairs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- compilers
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100