pytest-dev / pytest-dev/execnet

`remote_exec` hangs forever when remotely using `gevent` model only at python3

未关闭
#123 3 条评论 1 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

bug help wanted
主要语言
Python
星标
102
派生
47
PR 合并指标
30 天内没有已合并 PR

描述

The problem only occurs at python3.

Environment:

  • OS: Linux faa410b1f09b 4.19.121-linuxkit #1 SMP Thu Jan 21 15:36:34 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
  • Python version: Python 3.7.5

Dependencies:

  • gevent version: 21.1.2
  • execnet version: 1.8.0

Code:

import execnet
# hangs forever when remote is gevent
execnet.set_execmodel('thread', 'gevent')
gw = execnet.makegateway()
print(gw)
print(gw.remote_status())

print('channel.send')
print (gw.remote_exec("channel.send(333)").receive())
print('everything done')

Problem: Above code hangs forever.

Stdout:

<Gateway id='gw0' receive-live, thread model, 0 active channels>
<RInfo 'execmodel=gevent, numchannels=0, numexecuting=0'>
channel.send

Debug log:

[89653] gw0 sent <Message STATUS channel=1 lendata=0>
[89653] gw0 [receiver-thread] RECEIVERTHREAD: starting to run
[89655] creating workergateway on <execnet.gateway_base.Popen2IO object at 0x1106a5820>
[89655] gw0-worker [serve] spawning receiver thread
[89655] gw0-worker [serve] joining receiver thread
[89655] gw0-worker waiting for receiver thread to finish
[89655] gw0-worker [receiver-thread] RECEIVERTHREAD: starting to run
[89655] gw0-worker [receiver-thread] received <Message STATUS channel=1 lendata=0>
[89655] gw0-worker sent <Message CHANNEL_DATA channel=1 lendata=73>
[89653] gw0 [receiver-thread] received <Message CHANNEL_DATA channel=1 lendata=73>
[89653] gw0 [receiver-thread] received <Message CHANNEL_CLOSE channel=1 lendata=0>
[89655] gw0-worker sent <Message CHANNEL_CLOSE channel=1 lendata=0>
[89653] gw0 1 channel.__del__
[89653] gw0 sent <Message CHANNEL_EXEC channel=3 lendata=31>
[89655] gw0-worker [receiver-thread] received <Message CHANNEL_EXEC channel=3 lendata=31>

The code hangs while executing remote_exec. The problem only occurs when remote thread model is gevent.

I guess the problem occurred because while gevent is rapidly syncing with python3, execnet was not maintained :C.

Maybe related to function mapping when making ExecModel:
https://github.com/pytest-dev/execnet/blob/894533a515baf526c35ea0fe525b4866c6824ab8/execnet/gateway_base.py#L112-L125

I know this repo is unmaintained. Hoping for receiving someone's help!

贡献指南

这个仓库没有索引到贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

调研方向

使用 execnet 的 thread-to-gevent 配置以及 issue 中列出的版本,通过 Python 3 示例复现挂起问题。首先检查 execnet/gateway_base.py 中 112-125 行附近的 ExecModel 映射,然后跟踪 remote_exec 经过 receiver 和 worker 线程的过程。当远程调用返回 333,且程序在不挂起的情况下到达“everything done”时,即视为完成。

由索引模型根据 Issue 内容生成。

评估

技术栈
python
领域
distributed-systems
Issue 类型
缺陷
难度
4/5
预计耗时
3-5 天
活跃度
停滞
描述清晰度
基本清楚
新手友好度
35/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。