bigbio / bigbio/quantms

Optimisation of resources for the workflow

Open
#455 20 comments 0 reactions 3 assignees Claimed by @daichengxin View on GitHub
enhancement
Dominant language
Nextflow
Stars
83
Forks
57
Avg merge
4h 14m
Merged PRs (30d)
1

Description

### Description of feature

Currently, quantms have seven major categories for resources or processes:

```nextflow
withLabel:process_single {
cpus = { 1 }
memory = { 6.GB * task.attempt }
time = { 4.h * task.attempt }
}
withLabel:process_low {
cpus = { 4 * task.attempt }
memory = { 12.GB * task.attempt }
time = { 6.h * task.attempt }
}
withLabel:process_very_low {
cpus = { 2 * task.attempt}
memory = { 4.GB * task.attempt}
time = { 3.h * task.attempt}
}
withLabel:process_medium {
cpus = { 8 * task.attempt }
memory = { 36.GB * task.attempt }
time = { 8.h * task.attempt }
}
withLabel:process_high {
cpus = { 12 * task.attempt }
memory = { 72.GB * task.attempt }
time = { 16.h * task.attempt }
}
withLabel:process_long {
time = { 20.h * task.attempt }
}
withLabel:process_high_memory {
memory = { 200.GB * task.attempt }
}
```

However, some of my current analyses showed that resource usage, for example, for DIA analysis, could be optimized much more at the process level. See some results from my analyses.

### Dataset: PXD030304

**CPU Usage**:

![Screenshot 2024-12-05 at 09 08 27](https://github.com/user-attachments/assets/f5f1f412-d090-4da6-9dbe-147f01515b30)

**Memory Usage**:

![Screenshot 2024-12-05 at 09 08 43](https://github.com/user-attachments/assets/dca9cb4a-0f6b-463a-aa02-a5c1dff1193d)

**IO Usage**:

![Screenshot 2024-12-05 at 09 08 55](https://github.com/user-attachments/assets/bd0e53a6-1d78-4924-b860-a337cad235ed)

Most of the processes are under 50% of usage of memory and CPU which looks like a waist of resources?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.