DiamondLightSource / DiamondLightSource/httomo

Nonexistent method in pipeline YAML causes YAML checker to fail unexpectedly

Open
#331 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
10
Forks
5
Avg merge
5d 17h
Merged PRs (30d)
1

Description

Suppose the following pipeline YAML file is given to the YAML checker (this is a minimal example that reproduces the issue):
```yaml
- method: standard_tomo
module_path: httomo.data.hdf.loaders
parameters:
name: tomo
data_path: entry1/tomo_entry/data/data
image_key_path: entry1/tomo_entry/instrument/detector/image_key
rotation_angles:
data_path: /entry1/tomo_entry/data/rotation_angle
dimension: 1
pad: 0
- method: find_center_vol
module_path: httomolibgpu.recon.rotation
parameters:
ind: mid
smin: -50
smax: 50
srad: 6.0
step: 0.25
ratio: 0.5
drop: 20
```

and notice the incorrect method name `find_center_vol` (should be `find_center_vo`).

The YAML checker correctly reports this as being something incorrect in the YAML. However, afterwards, the YAML checker then fails abruptly:
```
Checking that the YAML_CONFIG is properly indented and has valid mappings and tags...
Sanity check of the YAML_CONFIG was successfully done...

Checking that the first method in the pipeline is a loader...
Loader check successful!!

'httomolibgpu.recon.rotation/find_center_vol' is not a valid method. Please recheck the yaml file.
Traceback (most recent call last):
File "/dls/science/users/twi18192/conda-envs/httomo/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/dls/science/users/twi18192/conda-envs/httomo/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/dls/science/users/twi18192/httomo/httomo/__main__.py", line 4, in
main()
File "/dls/science/users/twi18192/conda-envs/httomo/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
File "/dls/science/users/twi18192/conda-envs/httomo/lib/python3.10/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
File "/dls/science/users/twi18192/conda-envs/httomo/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/dls/science/users/twi18192/conda-envs/httomo/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/dls/science/users/twi18192/conda-envs/httomo/lib/python3.10/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "/dls/science/users/twi18192/httomo/httomo/cli.py", line 46, in check
return validate_yaml_config(yaml_config, in_data)
File "/dls/science/users/twi18192/httomo/httomo/yaml_checker.py", line 403, in validate_yaml_config
are_param_names_known = check_parameter_names_are_known(conf)
File "/dls/science/users/twi18192/httomo/httomo/yaml_checker.py", line 171, in check_parameter_names_are_known
template_yaml_conf = _get_template_yaml_conf(conf)
File "/dls/science/users/twi18192/httomo/httomo/yaml_checker.py", line 327, in _get_template_yaml_conf
tmp_conf = yaml_loader(f)
File "/dls/science/users/twi18192/httomo/httomo/ui_layer.py", line 280, in yaml_loader
with open(file_path, "r") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/dls/science/users/twi18192/httomo/yaml_templates/httomolibgpu/httomolibgpu.recon.rotation/find_center_vol.yaml'
```

The YAML checker runs _all_ checks, even if some fail, in order to be able to report as many issues in one go, to avoid the user having to constantly re-run the checker each time they fix one error.

The checks are the following: https://github.com/DiamondLightSource/httomo/blob/5ee850af2296f0c285681850035a7554a7347c14/httomo/yaml_checker.py#L396-L409

If a method is found to _not exist_ via the check `check_methods_exist_in_templates()` (ie, `do_methods_exist` gets assigned to `False`), the next check of if the parameter names in the method's config are known still runs.

Meaning, if a nonexistent method is in the pipeline (such as `find_center_vol`), then the YAML checker will carry on after reporting that the method doesn't exist, and attempt to open a YAML template file associated with the nonexistent method. This then obviously fails because a nonexistent method won't have a YAML template. This explains the `FileNotFoundError` at the end of the stacktrace:
```
FileNotFoundError: [Errno 2] No such file or directory: '/dls/science/users/twi18192/httomo/yaml_templates/httomolibgpu/httomolibgpu.recon.rotation/find_center_vol.yaml'
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.