common-workflow-language / common-workflow-language/cwltool

Runtime Error on large workflows with large number of threads/steps

Open
#1,347 3 comments 0 reactions 0 assignees View on GitHub
status/awaiting user response
Dominant language
Python
Stars
376
Forks
255
Avg merge
2d 7h
Merged PRs (30d)
12

Description

When running a large workflow with the `--parallel` flag that creates many branches / threads for a single step, the cwltool throws`RuntimeError: can't start new thread`

## Expected Behavior
The cwltool properly closes and disposes off previously started threads (and checks for available resources) before starting new threaded steps.

## Actual Behavior
The tool crashes, by trying to create too many jobs/threads.
It crashes after trying to create >1.2K threads. I am guessing that the resource calculation is a bit off.
On our linux systems the max thread/tasks limit is 12288. We can increase that limit but likely the tool would just crash at a higher threshold.

## Workflow Code
```
class: Workflow
cwlVersion: v1.0

requirements:
ResourceRequirement:
ramMin: 5000
coresMin: 8
SubworkflowFeatureRequirement: {}
ScatterFeatureRequirement: {}
StepInputExpressionRequirement: {}

inputs:

input_fasta: File
maskfile: File

outputs:
predicted_proteins:
type: File
outputSource: post-processing/predicted_proteins
predicted_seq:
type: File
outputSource: post-processing/predicted_seq

steps:

FGS:
in:
input_fasta: input_fasta
seq_type: { default: "0"}
output:
source: input_fasta
valueFrom: $(self.basename).fgs
out: [ predicted_proteins_out, predicted_proteins_ffn, predicted_proteins_faa ]
run: FragGeneScan/FGS.cwl

post-processing:
in:
masking_file: maskfile
predicted_proteins_fgs_out: FGS/predicted_proteins_out
predicted_proteins_fgs_ffn: FGS/predicted_proteins_ffn
predicted_proteins_fgs_faa: FGS/predicted_proteins_faa
basename:
source: input_fasta
valueFrom: $(self.basename)
out: [ predicted_proteins, predicted_seq ]
run: post-processing.cwl
```

Full workflow available here:
https://github.com/bio-ontology-research-group/ebi_pipeline/tree/pipeline_v5_local

Input sequence is here: ftp://ftp.sra.ebi.ac.uk/vol1/fastq/ERR318/ERR318583/ERR318583_1.fastq.gz

## Full Traceback
```
DEBUG Job "FGS_10660" cannot run yet, resources ({'cores': 4, 'ram': 5000, 'tmpdirSize': 1024, 'outdirSize': 1024}) are not available (already allocated ram is 600, allocated cores is 20, max ram 126091, max cores 20
DEBUG Job "FGS_10661" cannot run yet, resources ({'cores': 4, 'ram': 5000, 'tmpdirSize': 1024, 'outdirSize': 1024}) are not available (already allocated ram is 600, allocated cores is 20, max ram 126091, max cores 20
ERROR Unhandled error:
can't start new thread
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/cwltool/main.py", line 1132, in main
tool, initialized_job_order_object, runtimeContext, logger=_logger
File "/usr/local/lib/python3.6/dist-packages/cwltool/executors.py", line 59, in __call__
return self.execute(process, job_order_object, runtime_context, logger)
File "/usr/local/lib/python3.6/dist-packages/cwltool/executors.py", line 150, in execute
self.run_jobs(process, job_order_object, logger, runtime_context)
File "/usr/local/lib/python3.6/dist-packages/cwltool/executors.py", line 432, in run_jobs
self.run_job(job, runtime_context)
File "/usr/local/lib/python3.6/dist-packages/cwltool/executors.py", line 398, in run_job
thread.start()
File "/usr/lib/python3.6/threading.py", line 846, in start
_start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread
```

## Your Environment
* cwltool version: cwltool 3.0.20200807132242
* OS: reproduced on Ubuntu 16.04 and Ubuntu 18.04 64-bit

Thanks in advance.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.