MetOffice / MetOffice/CDDS

Overriding --gres=tmp requires two broadcast commands.

Open
#659 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
8
Forks
7
Avg merge
1d 15h
Merged PRs (30d)
23

Description

The problem is that the `--gres` directive value is set from the `MIP_CONVERT_TMP_SPACE` jinja2 value https://github.com/MetOffice/CDDS/blob/ac9ae76006c11f050072a94a920352e248b47334/cdds/cdds/workflows/conversion/rose-suite.conf#L32.

So if you try and raise the limit of the `--gres` directive with a `cylc broadcast`, the wrapper may still throw an error because because the `check_disk_usage` function will be passed the value from

https://github.com/MetOffice/CDDS/blob/ac9ae76006c11f050072a94a920352e248b47334/cdds/cdds/convert/mip_convert_wrapper/wrapper.py#L48

which won't have updated.

This can result in the following error.

```
Usage of $TMPDIR measured at 16739MB, which exceeds allocation of 8192MB
Usage of $TMPDIR measured at 16739MB, which exceeds allocation of 8192MB
Traceback (most recent call last):
File "{redacted}/conda_environments/cdds-3.3.0/lib/python3.10/site-packages/cdds/convert/command_line.py", line 109, in main_run_mip_convert
exit_code = run_mip_convert_wrapper()
File "{redacted}/conda_environments/cdds-3.3.0/lib/python3.10/site-packages/cdds/convert/mip_convert_wrapper/wrapper.py", line 129, in run_mip_convert_wrapper
check_disk_usage(staging_dir, max_temp_space_in_mb)
File "{redacted}/conda_environments/cdds-3.3.0/lib/python3.10/site-packages/cdds/convert/mip_convert_wrapper/actions.py", line 210, in check_disk_usage
raise MipConvertWrapperDiskUsageError(msg1)
cdds.convert.exceptions.MipConvertWrapperDiskUsageError: Usage of $TMPDIR measured at 16739MB, which exceeds allocation of 8192MB
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the --gres value in cdds/cdds/workflows/conversion/rose-suite.conf and the max_temp_space_in_mb handling in cdds/cdds/convert/mip_convert_wrapper/wrapper.py. Trace how cylc broadcast values reach run_mip_convert_wrapper and check_disk_usage in actions.py. Done means raising --gres with one broadcast no longer leaves the wrapper enforcing the old temporary-space limit.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, devops
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.