microsoft / microsoft/Power-Fx

Incorrect results for string functions on args with higher-plane characters

Open
#719 3 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Needs Discussion
Dominant language
C#
Stars
3.4k
Forks
358
Avg merge
10h 34m
Merged PRs (30d)
3

Description

Some functions are improperly treating strings that contain higher-plane Unicode characters (i.e., those with code points at U+10000 and higher, including most emoji) as if each of those characters were two characters long and unintelligible.

Split("🦓🦊🐺𐊀","")

Table({Value:"�"},{Value:"�"},{Value:"�"},{Value:"�"},{Value:"�"},{Value:"�"},{Value:"�"},{Value:"�"})
(Should be Table({Value:"🦓"},{Value:"🦊"},{Value:"🐺"},{Value:"𐊀"}))

Left("xyz🅰🅱🅲", 6)

"xyz🅰�" (note the ending character which is apparently half of the 🅱 emoji; this should just evaluate to the same string as was passed in)

Similarly, Right("🅰🅱🅲def", 6)"�🅲def", and Mid("🅰🅱🅲def", 2, 4)"�🅱�".

I believe something is going awry with how PowerFx is handling characters wider than 16 bits, and strings aren't being kept in a consistent translation format, which is leading to these errors.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the Split, Left, Right, and Mid examples from the issue with higher-plane Unicode characters. Trace the string handling used by these functions and add coverage for the shown cases; done means each function preserves complete Unicode characters without replacement characters or split surrogate pairs.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
compilers
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.