galaxyproject / galaxyproject/total-perspective-vortex
Add support for expressions in context variables
- Dominant language
- Python
- Stars
- 18
- Forks
- 19
- Avg merge
- 4d 9h
- Merged PRs (30d)
- 5
Description
### context
[From hrzolix on matrix](https://matrix.to/#/!rfLDbcWEWZapZrujix:gitter.im/$i4AdcVKJ5tqSSEmXd95tu7DapYcpwBsYTEXYh24sZsc?via=gitter.im&via=matrix.org)
[hrzolix]
I'm trying to give our users freely to choose parameters for tools, so far cores and memory work but I've tried to add the option for choosing queue, because we in background we have cpu, cpu_30, cpu_180 (7, 30, 180 days). I'm running into a problem here NameError: name 'params' is not defined
Here is the tpv_rules_local.yml:
```yaml
global:
default_inherits: default
tools:
default:
abstract: true
cores: 2
mem: 4
params:
queue: 'cpu_180'
rules:
- id: resource_params_defined
if: |
param_dict = job.get_param_values(app)
param_dict.get('__job_resource', {}).get('__job_resource__select') == 'yes'
cores: int(job.get_param_values(app)['__job_resource']['cores'])
mem: int(job.get_param_values(app)['__job_resource']['mem'])
params:
queue: "{str(job.get_param_values(app)['__job_resource']['queue'])}"
destinations:
local_env:
runner: local_runner
max_accepted_cores: 1
params:
tmp_dir: true
pbs:
runner: pbs_drmaa
params:
native_specification: "-q {params['queue']} -l select=1:ncpus={cores}:mem={mem}GB"
```
[Nuwan Goonasekera]
Can you try using a context variable instead? Using params within params is possible, but tricky and probably not worth the trouble. Something like this should work:
```yaml
global:
default_inherits: default
tools:
default:
abstract: true
cores: 2
mem: 4
context:
queue: 'cpu_180'
rules:
- id: resource_params_defined
if: |
param_dict = job.get_param_values(app)
param_dict.get('__job_resource', {}).get('__job_resource__select') == 'yes'
cores: int(job.get_param_values(app)['__job_resource']['cores'])
mem: int(job.get_param_values(app)['__job_resource']['mem'])
context:
queue: "{str(job.get_param_values(app)['__job_resource']['queue'])}"
destinations:
local_env:
runner: local_runner
max_accepted_cores: 1
params:
tmp_dir: true
pbs:
runner: pbs_drmaa
params:
native_specification: "-q {queue} -l select=1:ncpus={cores}:mem={mem}GB"
```
[hrzolix]
Hi Nuwan, I tried this and I got an error for queue does not exits, apparently it parses directly: -q {str(job.get_param_values(app)['__job_resource']['queue'])} -l select=1:ncpus=4:mem=8GB
[Nuwan Goonasekera]
Aah yes! I forgot that context variables must be constants. In that case, can you try reverting to your original code, but refer to params as entity.params?:
```yaml
pbs:
runner: pbs_drmaa
params:
native_specification: "-q {entity.params['queue']} -l select=1:ncpus={cores}:mem={mem}GB"
```
In the long run, we should probably consider adding support for computed context variables instead. That makes it much nicer I reckon
[hrzolix]
hm again its the same unfortunately : -q {str(job.get_param_values(app)['__job_resource']['queue'])} -l select=1:ncpus=6:mem=4GB
Here is the job_resource_params_conf.xml if it helps
```xml
CPU_180 (default)
CPU_30
CPU
```
[Nuwan Goonasekera]
You're right, it doesn't work that way. This is because of the evaluation order of variables. You can use this instead:
```yaml
global:
default_inherits: default
tools:
default:
abstract: true
cores: 2
mem: 4
env:
queue: 'cpu_180'
rules:
- id: resource_params_defined
if: |
param_dict = job.get_param_values(app)
param_dict.get('__job_resource', {}).get('__job_resource__select') == 'yes'
cores: int(job.get_param_values(app)['__job_resource']['cores'])
mem: int(job.get_param_values(app)['__job_resource']['mem'])
env:
queue: "{str(job.get_param_values(app)['__job_resource']['queue'])}"
destinations:
local_env:
runner: local_runner
max_accepted_cores: 1
params:
tmp_dir: true
pbs:
runner: pbs_drmaa
params:
native_specification: "-q {[env['value'] for env in env if env['name'] == 'queue'][0]} -l select=1:ncpus={cores}:mem={mem}GB"
```
That'll work for now, but the downside is that it passes a needless environment variable to the tool
Contributor guide
Assessment
This issue has not been assessed yet.