galaxyproject / galaxyproject/total-perspective-vortex
Resubmission with OOM doesn't seem to be kicked off
- Dominant language
- Python
- Stars
- 18
- Forks
- 19
- Avg merge
- 4d 9h
- Merged PRs (30d)
- 5
Description
On my k8s setup, having fixed the OOM labeling, I use the following tpv setup:
```
/galaxy/server/config/persist-seq-extra-tpv.yaml:
applyToJob: true
applyToWeb: true
applyToSetupJob: false
applyToWorkflow: true
applyToNginx: false
tpl: false
content: |
global:
default_inherits: default
tools:
default:
mem: 6
env:
OOM_TOOL_MEMORY: "{mem * int(job.destination_params.get('SCALING_FACTOR', 1)) if job.destination_params else 1}"
params:
SCALING_FACTOR: "{2 * int(job.destination_params.get('SCALING_FACTOR', 2)) if job.destination_params else 2}"
resubmit:
with_more_mem_on_failure:
condition: memory_limit_reached and attempt <= 3
destination: tpv_dispatcher
```
following I bit one of the test cases here.
on top of the existing one by default on the Helm chart:
```
job_conf.yml:
runners:
k8s:
k8s_extra_job_envs:
HDF5_USE_FILE_LOCKING: 'FALSE'
execution:
environments:
tpv_dispatcher:
tpv_config_files:
- https://raw.githubusercontent.com/galaxyproject/tpv-shared-database/main/tools.yml
- lib/galaxy/jobs/rules/tpv_rules_local.yml
- /galaxy/server/config/persist-seq-extra-tpv.yaml
```
on run I see the following DEBUG outputs from tpv:
```
tpv.core.entities DEBUG 2022-12-13 12:56:45,941 [pN:job_handler_0,p:8,tN:JobHandlerQueue.monitor_thread] Ranking destinations: [ id=k8s, cores=None, mem=None, gpus=None, env=None, params={'limits_cpu': '{cores}', 'limits_memory': '{mem}Gi', 'requests_cpu': '{cores}', 'requests_memory': '{mem}Gi'}, resubmit=None, tags= tags=[], rank=, inherits=None, context=None, rules={}] for entity: id=default, cores=1, mem=6, gpus=None, env={'OOM_TOOL_MEMORY': '1'}, params={'container_monitor': False, 'docker_default_container_id': 'quay.io/galaxyproject/galaxy-min:22.05', 'docker_enabled': 'true', 'tmp_dir': 'true', 'SCALING_FACTOR': '2'}, resubmit={}, tags= tags=[, ], rank=helpers.we, inherits=None, context={}, rules={} using custom function
```
which looks fine I guess besides the fact that `resubmit` is empty or "{}" depending on where you look. But then I see a more worrying:
```
galaxy.security.object_wrapper WARNING 2022-12-13 12:56:46,202 [pN:job_handler_0,p:8,tN:KubernetesRunner.work_thread-0] Unable to create dynamic subclass SafeStringWrapper(galaxy.model.none_like.None:,,,,,,,) for , None: type() doesn't support MRO entry resolution; use types.new_class()
```
which I suspect is related. On Galaxy, the jobs shows the OOM message on the UI ("Tool failed due to insufficient memory. Try with more memory."). Any idea of what might be going wrong? Is the DEBUG showing what one would expect? Thanks.
I also tried the setup in the readthedocs docs, it didn't work for me either, will post the results here as well.
Contributor guide
Assessment
This issue has not been assessed yet.