MetOffice / MetOffice/CDDS

Bug in `update_variables_from_validate`

Open
#1,100 0 comments 0 reactions 1 assignee Claimed by @mo-gill View on GitHub
bug
Dominant language
Python
Stars
8
Forks
7
Avg merge
1d 15h
Merged PRs (30d)
23

Description

Looks like there might be a bug in this tool.

Problem:
```
I'm using the 'update_variables_from_validate' to correct the error mentioned at [Common Issues - Climate Data Dissemination System](https://metoffice.github.io/CDDS/latest/operational_procedure/common_issues/#3-extract-validation-failure).

This produces a message saying "Using most recent log file validate_ap4_2026-09-07T0951Z.log". However, this isn't the most recent validate_ap4 log file - there is another one for 2026-09-09T0619Z.


Shell
ls -lt /data/users/cdds/cmip7_processing/john.rostron/proc/CMIP7/CMIP/UKCM2-0-LL_abrupt-4xCO2_r1i1p1f1/round-1/extract/log/validate_ap4*
-rw-r--r--. 1 john.rostron john.rostron 200028 Sep 9 09:45 /data/users/cdds/cmip7_processing/john.rostron/proc/CMIP7/CMIP/UKCM2-0-LL_abrupt-4xCO2_r1i1p1f1/round-1/extract/log/validate_ap4_2026-09-09T0619Z.log
-rw-r--r--. 1 john.rostron john.rostron 202371 Sep 7 14:21 /data/users/cdds/cmip7_processing/john.rostron/proc/CMIP7/CMIP/UKCM2-0-LL_abrupt-4xCO2_r1i1p1f1/round-1/extract/log/validate_ap4_2026-09-07T0951Z.log


Could you suggest how to resolve this? As a temporary fix, I could move the older log file out of the 'extract/log' directory temporarily whilst I run the update_variables_from_validate script?
```
TLDR; it's not picking up the latest log file from the directory

Have had a quick look, think this might be the cause:
https://github.com/MetOffice/CDDS/blob/96fdc97a319e0f7f654abb628a2f18882b431b7a/cdds/cdds/misc/update_variables_from_validate.py#L91-L93

It's likely just returning the last log in the iteration.

I'm going to adjust the docs until this is fixed to avoid more users encountering it, so that'll need changing once this is complete (see https://github.com/MetOffice/CDDS/pull/1101)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.