galaxyproject / galaxyproject/tpv-shared-database

Scale cores by memory

Open
#60 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
No language data
Stars
12
Forks
20
PR merge metrics
No merged PRs in 30d

Description

For some tools like Kraken2 I'd like to define the core count largely in terms of memory, but I also need to use different core counts depending on the destination. Most of the HPC systems I run on allocate memory strictly by core, so you get either 2GB/core or 4GB/core. If I set a static value for cores, I'm wasting cores.

And then occasionally I get allocated full nodes (either if the required memory is > 128 GB or on one particular HPC system that only allocates full nodes), in this case I may have more cores available than I actually want to use due to the slowdowns or increased memory consumption you get with excessively high core counts.

This results in things like [my kraken2 rule](https://github.com/galaxyproject/tpv-shared-database/issues/36#issuecomment-2045870817), the corresponding native spec for the 2GB/core system:

```
submit_native_specification: "--partition=RM-shared --time={time} --nodes=1 --ntasks={int(mem/2)} --mem={int(mem/2)*2000}"
```

where I also override `$GALAXY_SLOTS`:

```
env:
- name: GALAXY_SLOTS
value: "{min(int(mem/2), 64)}"
```

In general for any large memory tool, static core counts are fairly useless, and I may just prefer to always scale cores off of memory in the destination. I'm curious what others think.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.