blekhmanlab / blekhmanlab/compendium
Manager doesn't detect when pipeline errors crashed snakemake
- Dominant language
- Python
- Stars
- 9
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
If a job in the snakemake pipeline _fails_, the manager correctly identifies that something didn't complete. BUT, if there's an error from snakemake itself, the `running.txt` file never gets deleted, so the manager thinks it's running indefinitely. Example from project PRJNA530790:
```
[Fri Jan 13 19:11:12 2023]
Finished job 396.
269 of 446 steps (60%) done
Select jobs to execute...
[Fri Jan 13 19:11:12 2023]
rule sra_to_fastq:
input: SRR8849058/SRR8849058.sra
output: fastq/SRR8849058.fastq
jobid: 433
reason: Missing output files: fastq/SRR8849058.fastq; Input files updated by another job: SRR8849058/SRR8849058.sra
wildcards: sample=SRR8849058
threads: 4
resources: mem_mb=2000, mem_mib=1908, disk_mb=1000, disk_mib=954, tmpdir=, slurm_account=blekhman, slurm_partition=blekhman, runtime=480
WorkflowError:
SLURM job submission failed. The error message was sbatch: error: Batch job submission failed: Socket timed out on send/recv operation
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.