[SharePoint BUG] RefinementToken broken for accented values — Search REST API

Target SharePoint environment

SharePoint Online

What SharePoint development model, framework, SDK or API is this about?

SharePoint REST API

Developer environment

None

What browser(s) / client(s) have you tested

• 💥 Internet Explorer
• 💥 Microsoft Edge
• 💥 Google Chrome
• 💥 FireFox
• 💥 Safari
• mobile (iOS/iPadOS)
• mobile (Android)
• not applicable
• other (enter in the "Additional environment details" area below)

Additional environment details

• browser version
• SPFx version
• Node.js version
• etc

Describe the bug / error

Summary

The RefinementToken returned by /_api/search/postquery in RefinementResults does not work as-is in RefinementFilters when the refinement value contains non-ASCII characters such as accented characters.

The request returns 0 results silently, with no error.

Root cause

The API appears to be asymmetric in how it encodes and decodes refinement tokens:

• Token emission: the value is encoded using the hex representation of its UTF-8 bytes (é → c3a9)
• Filter matching: the token appears to be decoded using Latin-1 / character-code semantics (é → e9)

As a result, the RefinementToken returned by the API cannot be reused as-is when it contains a multi-byte UTF-8 character.

This does not affect pure ASCII values because, for characters below 0x80, UTF-8 and Latin-1/character-code representations are identical.

This explains why values such as Security work while Sécurité fails.

The same behavior can be reproduced directly through the SharePoint Search REST API, without any client-side application logic.

Tested and ruled out: NFC/NFD normalization, lowercase, diacritic stripping,Culture: 1036 (fr-FR), double UTF-8 encoding — all returned 0. Only the charcode re-encoding (or the string(mode="phrase") FQL fallback) works.

Impact

Any refinement filter (RefinableStringXX) on a non-ASCII value breaks if the raw API token is passed back — so most FR/DE/ES/etc. taxonomies are affected.

Fix applied on our side

Decodes the API token (hex UTF-8 → text), then re-encodes it as charcode hex before putting it in RefinementFilters. Falls back to string(mode="phrase") for characters outside Latin-1 (Arabic, CJK...).

Steps to reproduce

1. Create or use a SharePoint Online Managed Metadata field mapped to a refinable managed property, for example RefinableStringXX.

2. Add a taxonomy value containing an accented character, for example:
   Sécurité

3. Execute a Search REST API request using /_api/search/postquery with:

   {
     "request": {
       "Querytext": "*",
       "QueryTemplate": "{searchTerms} PromotedState:2",
       "Refiners": "RefinableStringXX",
       "RowLimit": 10
     }
   }
   

4. In the RefinementResults, locate the refinement corresponding to Sécurité.

5. Copy the RefinementToken returned by SharePoint. (starting by ǂǂ....)

6. Use the returned RefinementToken directly in RefinementFilters.

7. The request completes successfully but returns 0 results

Same request, only the encoding of é inside the token changes:
...437962657273c3a96375726... (UTF-8) → 0 results
...437962657273e96375726... (charcode) → 10 results ✔ (matches the expected RefinementCount)

Expected behavior

The RefinementToken returned by RefinementResults should be directly reusable in RefinementFilters.

A token returned by: RefinementResults should produce the corresponding refinement results when passed unchanged to:

RefinementFilters

Non-ASCII characters should be handled consistently between token generation and token matching.