python / python/cpython

matchfile Worker bug argument list too long when using matchfile to run many tests

未关闭
#139,207 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

tests type-bug
主要语言
Python
星标
77.2k
派生
35.9k
PR 合并指标
PR 指标待抓取

描述

Bug report

Bug description:

In https://github.com/python/cpython/issues/138313#issuecomment-3264096281, I'm trying to select different subsets of test_importlib to determine which ones are implicated in the failures. Unfortunately, it seems that when extracting the list of tests and then passing that to --matchfile the execution fails with:

+ ./python -u -W error -bb -E -m test --fast-ci --junit-xml test-results.xml test_importlib --matchfile cases.txt --dont-add-python-opts
Using random seed: 893560730
0:00:00 load avg: 0.54 Run 1 test in parallel using 1 worker process (timeout: 10 min, worker timeout: 15 min)
Warning -- regrtest worker thread failed: Traceback (most recent call last):
Warning --   File "/home/stan/cpython/Lib/test/libregrtest/run_workers.py", line 403, in run
Warning --     mp_result = self._runtest(test_name)
Warning --   File "/home/stan/cpython/Lib/test/libregrtest/run_workers.py", line 363, in _runtest
Warning --     retcode, tmp_files = self.run_tmp_files(worker_runtests,
Warning --                          ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
Warning --                                             stdout_file.fileno())
Warning --                                             ^^^^^^^^^^^^^^^^^^^^^
Warning --   File "/home/stan/cpython/Lib/test/libregrtest/run_workers.py", line 299, in run_tmp_files
Warning --     retcode = self._run_process(worker_runtests,
Warning --                                 stdout_fd, tmp_dir)
Warning --   File "/home/stan/cpython/Lib/test/libregrtest/run_workers.py", line 180, in _run_process
Warning --     popen = create_worker_process(runtests, output_fd, tmp_dir)
Warning --   File "/home/stan/cpython/Lib/test/libregrtest/worker.py", line 73, in create_worker_process
Warning --     return subprocess.Popen(cmd, **kwargs)
Warning --            ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
Warning --   File "/home/stan/cpython/Lib/subprocess.py", line 1037, in __init__
Warning --     self._execute_child(args, executable, preexec_fn, close_fds,
Warning --     ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Warning --                         pass_fds, cwd, env,
Warning --                         ^^^^^^^^^^^^^^^^^^^
Warning --     ...<5 lines>...
Warning --                         gid, gids, uid, umask,
Warning --                         ^^^^^^^^^^^^^^^^^^^^^^
Warning --                         start_new_session, process_group)
Warning --                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Warning --   File "/home/stan/cpython/Lib/subprocess.py", line 1969, in _execute_child
Warning --     raise child_exception_type(errno_num, err_msg, err_filename)
Warning -- OSError: [Errno 7] Argument list too long: '/home/stan/cpython/python'

== Tests result: FAILURE, WORKER BUG ==

It seems there's a limit to the number of lines that can be passed to a --matchfile.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

贡献指南

打开贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

调研方向

首先使用较大的 cases.txt 重现该命令,然后检查 Lib/test/libregrestest/run_workers.py 和 Lib/test/libregrestest/worker.py 中创建 worker 进程的相关部分。跟踪 --matchfile 选择如何传递到 subprocess.Popen,并将完成定义为在不出现 OSError: [Errno 7] Argument list too long 的情况下运行这个大型选择集。

由索引模型根据 Issue 内容生成。

评估

技术栈
python
领域
testing-qa
Issue 类型
缺陷
难度
3/5
预计耗时
1-2 天
活跃度
停滞
描述清晰度
基本清楚
新手友好度
45/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。