pytest-dev / pytest-dev/pluggy

Tracing crashes with UnicodeEncodeError on surrogate escapes in hook arguments/results

Open Beginner friendly
#681 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.7k
Forks
160
Avg merge
21h 4m
Merged PRs (30d)
6

Description

Problem

When pluggy tracing is enabled (e.g. pytest's --debug flag), hook arguments or return values containing surrogate escape characters (like \ud800) cause a UnicodeEncodeError crash. This is because _format_message in _tracing.py uses str() to format values, and the resulting string with literal surrogates cannot be written to most output targets.

Reported originally as pytest-dev/pytest#13750.

Analysis

_format_message currently formats two kinds of data identically using str():

  1. Structural labels passed as positional args — e.g. "finish", "-->", hook names — which are always safe ASCII strings
  2. Python values — hook kwargs values (rendered via the extra dict) and hook return values — which may contain arbitrary data including surrogates

The fix needs to apply repr() only to the value positions, not to structural labels.

Intended Solution

  1. In _tracing.py line 38 — use {value!r} for the extra/kwargs dict values:

    lines.append(f"{indent}    {name}: {value!r}\n")
    

    This makes kwargs values in trace output show their type (e.g. 'lfplugin' instead of lfplugin, PosixPath('/foo') instead of /foo) and safely escapes surrogates.

  2. In _manager.py line 506 — use repr() on the hook result before passing it as a trace arg:

    hooktrace("finish", hook_name, "-->", repr(outcome.get_result()))
    

    This ensures the result value is safely formatted, while "finish", hook_name, and "-->" remain plain str()-formatted (via _format_message's existing map(str, args)).

  3. Do NOT change the content = " ".join(map(str, args)) line in _format_message. Keeping str() there preserves readable structural output without quoting labels. The caller (_manager.py) is responsible for pre-formatting any unsafe values with repr().

This avoids the double-repr problem and keeps trace output readable:

  finish pytest_runtest_call --> '\ud800' [hook]
      config: <Config object at 0x...>
      plugin_name: 'lfplugin'

Rather than the over-quoted version that blanket repr() in _format_message would produce:

  'finish' 'pytest_runtest_call' '-->' "'\\ ud800'" [hook]

References

  • pytest-dev/pytest#13750
  • #627 (PR with discussion leading to this distilled approach)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with _format_message in _tracing.py around line 38 and the hook result tracing call in _manager.py around line 506. Verify tracing with surrogate-containing hook arguments and results no longer raises UnicodeEncodeError, while structural labels remain readable and values are safely represented.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
85/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.