galaxyproject / galaxyproject/total-perspective-vortex

Add support for expressions in context variables

Open
#120 0 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
18
Forks
19
Avg merge
4d 9h
Merged PRs (30d)
5

Description

### context

[From hrzolix on matrix](https://matrix.to/#/!rfLDbcWEWZapZrujix:gitter.im/$i4AdcVKJ5tqSSEmXd95tu7DapYcpwBsYTEXYh24sZsc?via=gitter.im&via=matrix.org)

[hrzolix]
I'm trying to give our users freely to choose parameters for tools, so far cores and memory work but I've tried to add the option for choosing queue, because we in background we have cpu, cpu_30, cpu_180 (7, 30, 180 days). I'm running into a problem here NameError: name 'params' is not defined

Here is the tpv_rules_local.yml:

```yaml
global:
default_inherits: default

tools:
default:
abstract: true
cores: 2
mem: 4
params:
queue: 'cpu_180'
rules:
- id: resource_params_defined
if: |
param_dict = job.get_param_values(app)
param_dict.get('__job_resource', {}).get('__job_resource__select') == 'yes'
cores: int(job.get_param_values(app)['__job_resource']['cores'])
mem: int(job.get_param_values(app)['__job_resource']['mem'])
params:
queue: "{str(job.get_param_values(app)['__job_resource']['queue'])}"

destinations:
local_env:
runner: local_runner
max_accepted_cores: 1
params:
tmp_dir: true
pbs:
runner: pbs_drmaa
params:
native_specification: "-q {params['queue']} -l select=1:ncpus={cores}:mem={mem}GB"
```

[Nuwan Goonasekera]
Can you try using a context variable instead? Using params within params is possible, but tricky and probably not worth the trouble. Something like this should work:
```yaml
global:
default_inherits: default

tools:
default:
abstract: true
cores: 2
mem: 4
context:
queue: 'cpu_180'
rules:
- id: resource_params_defined
if: |
param_dict = job.get_param_values(app)
param_dict.get('__job_resource', {}).get('__job_resource__select') == 'yes'
cores: int(job.get_param_values(app)['__job_resource']['cores'])
mem: int(job.get_param_values(app)['__job_resource']['mem'])
context:
queue: "{str(job.get_param_values(app)['__job_resource']['queue'])}"

destinations:
local_env:
runner: local_runner
max_accepted_cores: 1
params:
tmp_dir: true
pbs:
runner: pbs_drmaa
params:
native_specification: "-q {queue} -l select=1:ncpus={cores}:mem={mem}GB"
```

[hrzolix]
Hi Nuwan, I tried this and I got an error for queue does not exits, apparently it parses directly: -q {str(job.get_param_values(app)['__job_resource']['queue'])} -l select=1:ncpus=4:mem=8GB

[Nuwan Goonasekera]
Aah yes! I forgot that context variables must be constants. In that case, can you try reverting to your original code, but refer to params as entity.params?:
```yaml
pbs:
runner: pbs_drmaa
params:
native_specification: "-q {entity.params['queue']} -l select=1:ncpus={cores}:mem={mem}GB"
```

In the long run, we should probably consider adding support for computed context variables instead. That makes it much nicer I reckon

[hrzolix]
hm again its the same unfortunately : -q {str(job.get_param_values(app)['__job_resource']['queue'])} -l select=1:ncpus=6:mem=4GB
Here is the job_resource_params_conf.xml if it helps
```xml



CPU_180 (default)
CPU_30
CPU

```

[Nuwan Goonasekera]
You're right, it doesn't work that way. This is because of the evaluation order of variables. You can use this instead:
```yaml
global:
default_inherits: default

tools:
default:
abstract: true
cores: 2
mem: 4
env:
queue: 'cpu_180'
rules:
- id: resource_params_defined
if: |
param_dict = job.get_param_values(app)
param_dict.get('__job_resource', {}).get('__job_resource__select') == 'yes'
cores: int(job.get_param_values(app)['__job_resource']['cores'])
mem: int(job.get_param_values(app)['__job_resource']['mem'])
env:
queue: "{str(job.get_param_values(app)['__job_resource']['queue'])}"

destinations:
local_env:
runner: local_runner
max_accepted_cores: 1
params:
tmp_dir: true
pbs:
runner: pbs_drmaa
params:
native_specification: "-q {[env['value'] for env in env if env['name'] == 'queue'][0]} -l select=1:ncpus={cores}:mem={mem}GB"
```
That'll work for now, but the downside is that it passes a needless environment variable to the tool

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.