openai / openai/tiktoken

encode_with_unstable does not handle surrogate pairs like encode

Open
#541 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
19.3k
Forks
1.6k
PR merge metrics
No merged PRs in 30d

Description

Encoding.encode and Encoding.encode_ordinary already handle surrogate pairs/lone surrogates by retrying after UTF-16 surrogate repair.

However, Encoding.encode_with_unstable currently raises UnicodeEncodeError for the same inputs.

Repro:

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")

enc.encode("\ud83d\udc4d")  # works, same as "👍"
enc.encode_with_unstable("\ud83d\udc4d")  # raises UnicodeEncodeError

enc.encode("\ud83d")  # works, same as "�"
enc.encode_with_unstable("\ud83d")  # raises UnicodeEncodeError

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by comparing the encode_with_unstable implementation with Encoding.encode and Encoding.encode_ordinary, focusing on their handling of UTF-16 surrogate pairs and lone surrogates. Add regression coverage for the paired and lone-surrogate examples in the issue, then run the relevant test suite and confirm encode_with_unstable matches the existing methods without raising UnicodeEncodeError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.