pytest-dev / pytest-dev/pytest

Run pexpect tests on CI by default

Open
#12,884 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

type: selftests
Dominant language
Python
Stars
14.5k
Forks
3.4k
Avg merge
2d 9h
Merged PRs (30d)
35

Description

  • pytester tests that require pexpect are silently skipped if pexpect is unavailable:

https://github.com/pytest-dev/pytest/blob/f373974707f57a0b28d12563e4d03c7cd54c70d9/src/_pytest/pytester.py#L1513

  • We default to not installing pexpect even on CI, only having it as an optional dependency with a -pexpect factor:

https://github.com/pytest-dev/pytest/blob/f373974707f57a0b28d12563e4d03c7cd54c70d9/tox.ini#L79

  • The only CI environment where we include that is a Python 3.8 one:

https://github.com/pytest-dev/pytest/blob/f373974707f57a0b28d12563e4d03c7cd54c70d9/.github/workflows/test.yml#L116-L120

  • Yet, we have 40 (!) selftests that use pexpect - most notably, most of test_debugging.py (38 of 59 skipped)

  • And that caused us to miss #12497 on our own CI

Why is pexpect an optional factor at all? From what I can gather, that was introduced in bd8a2cc18c1898673665023556e73134e78e4d75 because "it doesn't install on Windows anymore" back in 2013.

Nowadays, it seems to be partially available for Windows, though not pexpect.spawn which we use for Pytester.spawn():

https://github.com/pytest-dev/pytest/blob/f373974707f57a0b28d12563e4d03c7cd54c70d9/src/_pytest/pytester.py#L1508-L1522

I think we should either:

  • Figure out if we can change Pytester in a backwards-compatible way to use the cross-platform PopenSpawn instead. Pexpect claims:

PopenSpawn is not a direct replacement for spawn. Many programs only offer interactive behaviour if they detect that they are running in a terminal. When run by PopenSpawn, they may behave differently.

  • Or if not, at least install pexpect unconditionally on Linux (and perhaps macOS) as part of our dev dependencies, using a environment marker (added in pip 6.0 in 2014, so that wasn't an option back when the tox factor was introduced).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read tox.ini and .github/workflows/test.yml to understand the current pexpect factor and CI matrix, then inspect src/_pytest/pytester.py and testing/test_debugging.py for the skipped cases. Determine whether the existing spawn behavior can be supported cross-platform or whether pexpect should be installed on supported CI systems; done means the relevant tests run by default without inappropriate skips.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ci-cd, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.