galaxyproject / galaxyproject/total-perspective-vortex

Resubmission with OOM doesn't seem to be kicked off

Open
#65 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
18
Forks
19
Avg merge
4d 9h
Merged PRs (30d)
5

Description

On my k8s setup, having fixed the OOM labeling, I use the following tpv setup:

```
/galaxy/server/config/persist-seq-extra-tpv.yaml:
applyToJob: true
applyToWeb: true
applyToSetupJob: false
applyToWorkflow: true
applyToNginx: false
tpl: false
content: |
global:
default_inherits: default
tools:
default:
mem: 6
env:
OOM_TOOL_MEMORY: "{mem * int(job.destination_params.get('SCALING_FACTOR', 1)) if job.destination_params else 1}"
params:
SCALING_FACTOR: "{2 * int(job.destination_params.get('SCALING_FACTOR', 2)) if job.destination_params else 2}"
resubmit:
with_more_mem_on_failure:
condition: memory_limit_reached and attempt <= 3
destination: tpv_dispatcher
```

following I bit one of the test cases here.

on top of the existing one by default on the Helm chart:

```
job_conf.yml:
runners:
k8s:
k8s_extra_job_envs:
HDF5_USE_FILE_LOCKING: 'FALSE'
execution:
environments:
tpv_dispatcher:
tpv_config_files:
- https://raw.githubusercontent.com/galaxyproject/tpv-shared-database/main/tools.yml
- lib/galaxy/jobs/rules/tpv_rules_local.yml
- /galaxy/server/config/persist-seq-extra-tpv.yaml
```

on run I see the following DEBUG outputs from tpv:

```
tpv.core.entities DEBUG 2022-12-13 12:56:45,941 [pN:job_handler_0,p:8,tN:JobHandlerQueue.monitor_thread] Ranking destinations: [ id=k8s, cores=None, mem=None, gpus=None, env=None, params={'limits_cpu': '{cores}', 'limits_memory': '{mem}Gi', 'requests_cpu': '{cores}', 'requests_memory': '{mem}Gi'}, resubmit=None, tags= tags=[], rank=, inherits=None, context=None, rules={}] for entity: id=default, cores=1, mem=6, gpus=None, env={'OOM_TOOL_MEMORY': '1'}, params={'container_monitor': False, 'docker_default_container_id': 'quay.io/galaxyproject/galaxy-min:22.05', 'docker_enabled': 'true', 'tmp_dir': 'true', 'SCALING_FACTOR': '2'}, resubmit={}, tags= tags=[, ], rank=helpers.we, inherits=None, context={}, rules={} using custom function
```

which looks fine I guess besides the fact that `resubmit` is empty or "{}" depending on where you look. But then I see a more worrying:

```
galaxy.security.object_wrapper WARNING 2022-12-13 12:56:46,202 [pN:job_handler_0,p:8,tN:KubernetesRunner.work_thread-0] Unable to create dynamic subclass SafeStringWrapper(galaxy.model.none_like.None:,,,,,,,) for , None: type() doesn't support MRO entry resolution; use types.new_class()
```

which I suspect is related. On Galaxy, the jobs shows the OOM message on the UI ("Tool failed due to insufficient memory. Try with more memory."). Any idea of what might be going wrong? Is the DEBUG showing what one would expect? Thanks.

I also tried the setup in the readthedocs docs, it didn't work for me either, will post the results here as well.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.