pytest-dev / pytest-dev/pytest

`UnicodeEncodeError` in pluggy with surrogate escape in parametrization and `--debug`

Open
#13,750 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

topic: collection topic: parametrize
Dominant language
Python
Stars
14.5k
Forks
3.4k
Avg merge
2d 9h
Merged PRs (30d)
35

Description

On Linux, with the current git main of pytest (8.5.0.dev60+g67738403d), pluggy 1.6.0 and Python 3.13.1, this test runs fine:

import pytest

@pytest.mark.parametrize("s", ["\ud800"])
def test_x(s):
    pass

but, somewhat contrary to its name, --debug=debug.log causes a bug instead:

_____________________ ERROR collecting test_surrogate.py _____________________
.venv/lib/python3.13/site-packages/pluggy/_hooks.py:512: in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/pluggy/_manager.py:120: in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/pluggy/_manager.py:475: in traced_hookexec
    return outcome.get_result()
           ^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/pluggy/_manager.py:472: in <lambda>
    lambda: oldcall(hook_name, hook_impls, caller_kwargs, firstresult)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
src/_pytest/python.py:240: in pytest_pycollect_makeitem
    return list(collector._genfunctions(name, obj))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
src/_pytest/python.py:466: in _genfunctions
    self.ihook.pytest_generate_tests.call_extra(methods, dict(metafunc=metafunc))
.venv/lib/python3.13/site-packages/pluggy/_hooks.py:573: in call_extra
    return self._hookexec(self.name, hookimpls, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/pluggy/_manager.py:120: in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/pluggy/_manager.py:475: in traced_hookexec
    return outcome.get_result()
           ^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/pluggy/_manager.py:472: in <lambda>
    lambda: oldcall(hook_name, hook_impls, caller_kwargs, firstresult)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
src/_pytest/python.py:114: in pytest_generate_tests
    metafunc.parametrize(*marker.args, **marker.kwargs, _param_mark=marker)
src/_pytest/python.py:1274: in parametrize
    ids = self._resolve_parameter_set_ids(
src/_pytest/python.py:1395: in _resolve_parameter_set_ids
    return id_maker.make_unique_parameterset_ids()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
src/_pytest/python.py:902: in make_unique_parameterset_ids
    resolved_ids = list(self._resolve_ids())
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
src/_pytest/python.py:945: in _resolve_ids
    yield "-".join(
src/_pytest/python.py:946: in <genexpr>
    self._idval(val, argname, idx)
src/_pytest/python.py:957: in _idval
    idval = self._idval_from_hook(val, argname)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
src/_pytest/python.py:985: in _idval_from_hook
    id: str | None = self.config.hook.pytest_make_parametrize_id(
.venv/lib/python3.13/site-packages/pluggy/_hooks.py:512: in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/pluggy/_manager.py:120: in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/pluggy/_manager.py:470: in traced_hookexec
    before(hook_name, hook_impls, caller_kwargs)
.venv/lib/python3.13/site-packages/pluggy/_manager.py:495: in before
    hooktrace(hook_name, kwargs)
.venv/lib/python3.13/site-packages/pluggy/_tracing.py:69: in __call__
    self.root._processmessage(self.tags, args)
.venv/lib/python3.13/site-packages/pluggy/_tracing.py:44: in _processmessage
    self._writer(self._format_message(tags, args))
E   UnicodeEncodeError: 'utf-8' codec can't encode character '\ud800' in position 140: surrogates not allowed

Reporting here as it's somewhat unclear whether a proper fix for this should be in pytest or in pluggy (possibly the latter though, given that it can probably be reproduced with any hook execution with a string argument that has surrogate escapes?).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the parametrized test with --debug=debug.log on the stated Python and pytest versions. Start with src/_pytest/python.py and the shown pluggy tracing paths, then determine which project owns handling surrogate-containing hook arguments; done when debug logging no longer raises UnicodeEncodeError and the regression is covered.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.