mlcommons / mlcommons/mlcube

SPython incompatible with windows. MLCube fails at import time

Open
#292 1 comment 0 reactions 1 assignee View on GitHub

@sergey-serebryakov is already working on this.

Since Feb 13, 2023.

Dominant language
Python
Stars
160
Forks
31
PR merge metrics
No merged PRs in 30d

Description

Trying to run a docker MLCube on windows fails because of unsupported imports. This happens because spython, the singularity python API, tries to import pwd, which doesn't exist on Windows. Here's the traceback:

$ mlcube run --mlcube=C:\Users\aleja\.medperf\localhost_8000\cubes\1\mlcube.yaml --task=prepare --platform=docker data_path="C:\Users\aleja\.medperf\localhost_8000\demo\137950e4e7b8de3baa4a9982c3ebea6a52bd33d3\mock_chexpert" labels_path="C:\Users\aleja\.medperf\localhost_8000\demo\137950e4e7b8de3baa4a9982c3ebea6a52bd33d3\mock_chexpert" output_path="C:\Users\aleja\.medperf\localhost_8000\data\tmp_26312981032654\data" Ptasks.prepare.parameters.input.data_path.opts="ro" Ptasks.prepare.parameters.input.labels_path.opts="ro"
Traceback (most recent call last):
  File "d:\users\aleja\anaconda3\envs\factored\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "d:\users\aleja\anaconda3\envs\factored\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "D:\Users\aleja\anaconda3\envs\factored\Scripts\mlcube.exe\__main__.py", line 7, in <module>
  File "d:\users\aleja\anaconda3\envs\factored\lib\site-packages\click\core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "d:\users\aleja\anaconda3\envs\factored\lib\site-packages\click\core.py", line 782, in main
    rv = self.invoke(ctx)
  File "d:\users\aleja\anaconda3\envs\factored\lib\site-packages\click\core.py", line 1256, in invoke
    Command.invoke(self, ctx)
  File "d:\users\aleja\anaconda3\envs\factored\lib\site-packages\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "d:\users\aleja\anaconda3\envs\factored\lib\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "d:\users\aleja\anaconda3\envs\factored\lib\site-packages\mlcube\__main__.py", line 119, in cli
    _ = SystemSettings().update_installed_runners()
  File "d:\users\aleja\anaconda3\envs\factored\lib\site-packages\mlcube\system_settings.py", line 111, in update_installed_runners
    installed_runners: t.Dict = Platform.get_installed_runners()
  File "d:\users\aleja\anaconda3\envs\factored\lib\site-packages\mlcube\platform.py", line 60, in get_installed_runners
    runner_cls: t.Type[Runner] = get_runner_class()
  File "d:\users\aleja\anaconda3\envs\factored\lib\site-packages\mlcube_singularity\__init__.py", line 40, in get_runner_class
    from mlcube_singularity.singularity_run import SingularityRun
  File "d:\users\aleja\anaconda3\envs\factored\lib\site-packages\mlcube_singularity\singularity_run.py", line 6, in <module>
    from spython.utils.terminal import (
  File "d:\users\aleja\anaconda3\envs\factored\lib\site-packages\spython\utils\__init__.py", line 5, in <module>
    from .terminal import (
  File "d:\users\aleja\anaconda3\envs\factored\lib\site-packages\spython\utils\terminal.py", line 12, in <module>
    import pwd
ModuleNotFoundError: No module named 'pwd'

This issue disappears when uninstall mlcube-singularity. It would be useful to provide some hints to the user if this error happens. Is singularity even supported on windows without WSL? If not, raising an error when installing mlcube-singularity could be another approach

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.