PyO3 / PyO3/pyo3

Problem with default sys.stdout.encoding producing UnicodeEncodeError

Open
#2,940 10 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
16.2k
Forks
1k
Avg merge
2d 6h
Merged PRs (30d)
66

Description

I just had this really weird bug, where no matter which Rust String I wanted to use in Python always raised this error when trying to print or convert the String: "UnicodeEncodeError('ascii', 'asdf´´', 4, 6, 'ordinal not in range(128)')"

I just now found the problem which caused this error to occur consistently:
The shell in which the rust executable was running didn't have the "LANG" environment variable set. This makes the python interpreter choose the default encoding "ASCII" which is (among others) used for output encoding. This error is easily overlooked because everything works fine if you only use ASCII characters (because UTF-8 is compatible with ASCII in that range). It only fails when using a character that is not included in ASCII, i.e. has a value of over 127.

Rust Strings are (by default) stored as utf-8 encoded bytes and the conversion between a Rust and a Python String apparently directly hands a pointer to the utf-8 encoded byte-array to the Python interpreter to produce a Python String.

Feature Request: Somehow tell the user that the Python output encoding should be UTF-8 (and this means properly setting the "LANG" variable), because otherwise this weird error might occur when dealing with special characters.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names no files or tests. Start by reproducing the UnicodeEncodeError with LANG unset, then inspect how PyO3 initializes Python and converts Rust strings for output. Done means users receive guidance or a reliable UTF-8 configuration instead of the unexplained encoding failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
api, backend
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.