blekhmanlab / blekhmanlab/compendium

Manager doesn't detect when pipeline errors crashed snakemake

Open
#14 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
9
Forks
0
PR merge metrics
No merged PRs in 30d

Description

If a job in the snakemake pipeline _fails_, the manager correctly identifies that something didn't complete. BUT, if there's an error from snakemake itself, the `running.txt` file never gets deleted, so the manager thinks it's running indefinitely. Example from project PRJNA530790:

```
[Fri Jan 13 19:11:12 2023]
Finished job 396.
269 of 446 steps (60%) done
Select jobs to execute...

[Fri Jan 13 19:11:12 2023]
rule sra_to_fastq:
input: SRR8849058/SRR8849058.sra
output: fastq/SRR8849058.fastq
jobid: 433
reason: Missing output files: fastq/SRR8849058.fastq; Input files updated by another job: SRR8849058/SRR8849058.sra
wildcards: sample=SRR8849058
threads: 4
resources: mem_mb=2000, mem_mib=1908, disk_mb=1000, disk_mib=954, tmpdir=, slurm_account=blekhman, slurm_partition=blekhman, runtime=480

WorkflowError:
SLURM job submission failed. The error message was sbatch: error: Batch job submission failed: Socket timed out on send/recv operation
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.