ipython / ipython/ipyparallel

Option of defining an environment script for starting engines (with SSHEngineSetLauncher)

オープン
#1,029 コメント 9 件 リアクション 0 件 担当者 0 名 GitHub で見る
主要言語
Jupyter Notebook
スター
2.6k
フォーク
1k
PR マージ指標
30日以内にマージされた PR はありません

説明

We need to adapt the environment before starting the actual ipp _engine_ (through the `SSHEngineSetLauncher`). Adapting the environment via the `environment` parameter (in class `SSHLauncher`) is possible, but it's not very practical, since the launcher needs to know all the engine computer details in advance.
Calling a local environment script before starting the _engine_ would give much more flexibility and would easily allow starting the engine e.g. in a python virtual environment, conda environment, etc.

Although the general idea is quite simple the actual implement is not as easy. The problem: one cannot simply start a shell, next the environment script and then the actual command because the process handle (in python) would point to the shell and not the actual command. Hence, the retrieved process id would be incorrect. Using some system code it would be still possible to retrieve the true process id, but this is especially difficult in windows which is why I tested an alternative solution:
Starting a shell, outputting all environment variable, running the environment script and outputting all environment variable again. A bit of python code analyses the output and collects the environment variable changes (especially added and/or changed env.vars). I have implemented a small test script that works under linux and windows.
The collected changes can then be easily passed to the `env` parameter of `subprocess.Popen` function.

Does this sound like an appropriate idea and solution?
As before I'm happy to provide the implementation, tests, documentation, etc.

コントリビューションガイド

コントリビューションガイドを開く

調査の方向性

Start by reading SSHEngineSetLauncher and SSHLauncher, then inspect the small cross-platform test script described in the issue and the subprocess.Popen environment handling. Done means an environment script can prepare the engine environment on Linux and Windows without making the Python process handle point to the shell, with implementation tests and documentation.

索引モデルが issue の本文から書いたものです。

評価

技術スタック
python, shell
領域
distributed-systems
issue の種類
機能追加
難易度
4/5
見積もり時間
3〜5日
活発さ
静か
明瞭さ
おおむね明確
初心者へのやさしさ
48/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。