llvm / llvm/torch-mlir

running tests on large number of cpu/cores causes resource exhaustion

Open
#1,302 2 comments 0 reactions 1 assignee View on GitHub

@powderluv is already working on this.

Since Aug 28, 2022.

Dominant language
C++
Stars
1.9k
Forks
736
Avg merge
5d 22h
Merged PRs (30d)
15

Description

When running on systems with 64+ cores you can run into issues with tests trying to spawn cpu_count * 1.1 threads.

https://github.com/llvm/torch-mlir/blob/f245613b71b82eb2ad7ead22ef3499ebcd925a92/python/torch_mlir_e2e_test/torchscript/framework.py#L334


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/anush/github/torch-mlir/build/tools/torch-mlir/python_packages/torch_mlir/torch_mlir_e2e_test/torchscript/framework.py", line 374, in worker
    compile_and_run_test(tests_dict[test_name], config))
  File "/home/anush/github/torch-mlir/build/tools/torch-mlir/python_packages/torch_mlir/torch_mlir_e2e_test/torchscript/framework.py", line 301, in compile_and_run_test
    trace = config.run(compiled, golden_trace)
  File "/home/anush/github/torch-mlir/build/tools/torch-mlir/python_packages/torch_mlir/torch_mlir_e2e_test/torchscript/configs/eager_mode.py", line 59, in run
    outps = attr(*inps)
  File "/home/anush/github/torch-mlir/build/tools/torch-mlir/python_packages/torch_mlir/torch_mlir_e2e_test/test_suite/rng.py", line 222, in forward
    torch.flatten(torch.std(b)),
  File "/home/anush/github/torch-mlir/mlir_venv/lib/python3.10/site-packages/torch/_tensor.py", line 1265, in __torch_function__
    ret = func(*args, **kwargs)
  File "/home/anush/github/torch-mlir/build/tools/torch-mlir/python_packages/torch_mlir/torch_mlir/eager_mode/torch_mlir_tensor.py", line 160, in __torch_dispatch__
    op_mlir_backend_callable = backend.compile(eager_module)
  File "/home/anush/github/torch-mlir/build/tools/torch-mlir/python_packages/torch_mlir/torch_mlir_e2e_test/eager_backends/refbackend.py", line 68, in compile
    run_pipeline_with_repro_report(
  File "/home/anush/github/torch-mlir/build/tools/torch-mlir/python_packages/torch_mlir/torch_mlir/compiler_utils.py", line 47, in run_pipeline_with_repro_report
    pm.run(module)
KeyboardInterrupt
Process ForkProcess-7:
Traceback (most recent call last):
  File "/home/anush/github/torch-mlir/build/tools/torch-mlir/python_packages/torch_mlir/torch_mlir/compiler_utils.py", line 47, in run_pipeline_with_repro_report
    pm.run(module)
RuntimeError: Failure while executing pass pipeline.

During handling of the above exception, another exception occurred:


We should cap it to max of 16. Will send a PR.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.