Climate-REF / Climate-REF/climate-ref
enso-characteristic diagnostic fails to copy successful output to results directory
- Dominant language
- Python
- Stars
- 26
- Forks
- 14
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 36
Description
## Describe the bug
The `enso-characteristics` diagnostic fails to copy the successful output from `scratch/` to `results/`, when running REF with a preliminary CMIP7 model dataset. Error message below (and attached).
Note that it runs successfully for a CMIP6 dataset (CanESM5-1 p1)
```
2026-08-18 22:20:59.254 +00:00 e0:03:42.870143s | INFO | SpawnProcess-1:263332 | climate_ref_core.providers:run:642 - Command execution successful
2026-08-18 22:20:59.258 +00:00 e0:03:42.873952s | ERROR | SpawnProcess-1:263332 | climate_ref_core.logging:redirect_logs:221 - Execution failed
Traceback (most recent call last):
File "", line 1, in
File "/fs/site7/eccc/crd/cccma/users/scrd115/code/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/multiprocessing/spawn.py", line 122, in spawn_main
exitcode = _main(fd, parent_sentinel)
│ │ └ 4
│ └ 16
└
File "/fs/site7/eccc/crd/cccma/users/scrd115/code/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/multiprocessing/spawn.py", line 135, in _main
return self._bootstrap(parent_sentinel)
│ │ └ 4
│ └
└
File "/fs/site7/eccc/crd/cccma/users/scrd115/code/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
self.run()
│ └
└
File "/fs/site7/eccc/crd/cccma/users/scrd115/code/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
│ │ │ │ │ └ {}
│ │ │ │ └
│ │ │ └ (,
│ └
└
File "/fs/site7/eccc/crd/cccma/users/scrd115/code/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/concurrent/futures/process.py", line 264, in _process_worker
r = call_item.fn(*call_item.args, **call_item.kwargs)
│ │ │ │ │ └ {'definition': ExecutionDefinition(diagnostic=ENSOCharacteristics(name='ENSO Characteristics'), key='cmip7_g127_CanESM5-1_r10...
│ │ │ │ └
│ │ │ └ ()
│ │ └
│ └
└
File "/fs/site7/eccc/crd/cccma/users/scrd115/REF/REF_v0.16.2/climate-ref/packages/climate-ref/src/climate_ref/executor/local.py", line 52, in _process_run
return execute_locally(definition=definition, log_level=log_level)
│ │ └ 'ERROR'
│ └ ExecutionDefinition(diagnostic=ENSOCharacteristics(name='ENSO Characteristics'), key='cmip7_g127_CanESM5-1_r10i1p2f1', datase...
└
File "/fs/site7/eccc/crd/cccma/users/scrd115/REF/REF_v0.16.2/climate-ref/packages/climate-ref-core/src/climate_ref_core/executor.py", line 65, in execute_locally
with redirect_logs(definition, log_level):
│ │ └ 'ERROR'
│ └ ExecutionDefinition(diagnostic=ENSOCharacteristics(name='ENSO Characteristics'), key='cmip7_g127_CanESM5-1_r10i1p2f1', datase...
└
File "/fs/site7/eccc/crd/cccma/users/scrd115/code/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 158, in __exit__
self.gen.throw(value)
│ │ │ └ FileNotFoundError(2, 'No such file or directory')
│ │ └
│ └
└
> File "/fs/site7/eccc/crd/cccma/users/scrd115/REF/REF_v0.16.2/climate-ref/packages/climate-ref-core/src/climate_ref_core/logging.py", line 219, in redirect_logs
yield
File "/fs/site7/eccc/crd/cccma/users/scrd115/REF/REF_v0.16.2/climate-ref/packages/climate-ref-core/src/climate_ref_core/executor.py", line 66, in execute_locally
return definition.diagnostic.run(definition=definition)
│ │ └ ExecutionDefinition(diagnostic=ENSOCharacteristics(name='ENSO Characteristics'), key='cmip7_g127_CanESM5-1_r10i1p2f1', datase...
│ └
└ ExecutionDefinition(diagnostic=ENSOCharacteristics(name='ENSO Characteristics'), key='cmip7_g127_CanESM5-1_r10i1p2f1', datase...
File "/fs/site7/eccc/crd/cccma/users/scrd115/REF/REF_v0.16.2/climate-ref/packages/climate-ref-core/src/climate_ref_core/diagnostics.py", line 637, in run
return self.build_execution_result(definition)
│ │ └ ExecutionDefinition(diagnostic=ENSOCharacteristics(name='ENSO Characteristics'), key='cmip7_g127_CanESM5-1_r10i1p2f1', datase...
│ └
└ ENSOCharacteristics(name='ENSO Characteristics')
File "/fs/site7/eccc/crd/cccma/users/scrd115/REF/REF_v0.16.2/climate-ref/packages/climate-ref-esmvaltool/src/climate_ref_esmvaltool/diagnostics/base.py", line 455, in build_execution_result
metric_bundle, output_bundle = self.format_result(
│ └ )>
└ ENSOCharacteristics(name='ENSO Characteristics')
File "/fs/site7/eccc/crd/cccma/users/scrd115/REF/REF_v0.16.2/climate-ref/packages/climate-ref-esmvaltool/src/climate_ref_esmvaltool/diagnostics/enso.py", line 449, in format_result
metrics = pd.read_csv(
│ └
└
File "/fs/site7/eccc/crd/cccma/users/scrd115/envs/ref_v0.16.2/lib/python3.12/site-packages/pandas/io/parsers/readers.py", line 873, in read_csv
return _read(filepath_or_buffer, kwds)
│ │ └ {'delimiter': ',', 'header': 'infer', 'names': ['dataset', 'metric_name', 'metric_value'], 'index_col': None, 'usecols': None...
│ └ PosixPath('/fs/site7/eccc/crd/cccma/users/scrd115/REF/REF_v0.16.2/climate-ref/.ref/scratch/esmvaltool/enso-characteristics/g1...
└
File "/fs/site7/eccc/crd/cccma/users/scrd115/envs/ref_v0.16.2/lib/python3.12/site-packages/pandas/io/parsers/readers.py", line 300, in _read
parser = TextFileReader(filepath_or_buffer, **kwds)
│ │ └ {'delimiter': ',', 'header': 'infer', 'names': ['dataset', 'metric_name', 'metric_value'], 'index_col': None, 'usecols': None...
│ └ PosixPath('/fs/site7/eccc/crd/cccma/users/scrd115/REF/REF_v0.16.2/climate-ref/.ref/scratch/esmvaltool/enso-characteristics/g1...
└
File "/fs/site7/eccc/crd/cccma/users/scrd115/envs/ref_v0.16.2/lib/python3.12/site-packages/pandas/io/parsers/readers.py", line 1645, in __init__
self._engine = self._make_engine(f, self.engine)
│ │ │ │ │ └ 'c'
│ │ │ │ └
│ │ │ └ PosixPath('/fs/site7/eccc/crd/cccma/users/scrd115/REF/REF_v0.16.2/climate-ref/.ref/scratch/esmvaltool/enso-characteristics/g1...
│ │ └
│ └
└
File "/fs/site7/eccc/crd/cccma/users/scrd115/envs/ref_v0.16.2/lib/python3.12/site-packages/pandas/io/parsers/readers.py", line 1904, in _make_engine
self.handles = get_handle(
│ │ └
│ └ None
└
File "/fs/site7/eccc/crd/cccma/users/scrd115/envs/ref_v0.16.2/lib/python3.12/site-packages/pandas/io/common.py", line 930, in get_handle
handle = open(
FileNotFoundError: [Errno 2] No such file or directory: '/fs/site7/eccc/crd/cccma/users/scrd115/REF/REF_v0.16.2/climate-ref/.ref/scratch/esmvaltool/enso-characteristics/g127_CanESM5-1_r10i1p2f1_g3062_v1_3e18a899/7389/executions/recipe_20260818_221807/work/diagnostic_metrics/plot_script/matrix.csv'
2026-08-18 22:20:59.489 +00:00 e0:03:43.105254s | INFO | SpawnProcess-1:263332 | climate_ref_core.logging:redirect_logs:224 - Diagnostic execution complete. Results available in esmvaltool/enso-characteristics/g127_CanESM5-1_r10i1p2f1_g3062_v1_3e18a899/7389
2026-08-18 22:20:59.493 +00:00 e0:03:43.109139s | ERROR | SpawnProcess-1:263332 | climate_ref_core.executor:execute_locally:77 - System error running 'esmvaltool/enso-characteristics/cmip7_g127_CanESM5-1_r10i1p2f1' (will be retried on next solve): [Errno 2] No such file or directory: '/fs/site7/eccc/crd/cccma/users/scrd115/REF/REF_v0.16.2/climate-ref/.ref/scratch/esmvaltool/enso-characteristics/g127_CanESM5-1_r10i1p2f1_g3062_v1_3e18a899/7389/executions/recipe_20260818_221807/work/diagnostic_metrics/plot_script/matrix.csv'
```
## System
- OS: Linux
- Python 3.12.13
```
provider version conda environment data path
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
example 0.16.2
esmvaltool 0.16.2 /fs/site7/eccc/crd/cccma/users/scrd115/REF/REF_v0.16.2/climate-ref/.ref/software/conda/e… /fs/site7/eccc/crd/cccma/users/scrd115/REF/REF_v0.16.2/climate-ref/.ref/cache/esmvaltool
ilamb 0.16.2 /fs/site7/eccc/crd/cccma/users/scrd115/REF/REF_v0.16.2/climate-ref/.ref/cache/ilamb
pmp 0.16.2 /fs/site7/eccc/crd/cccma/users/scrd115/REF/REF_v0.16.2/climate-ref/.ref/software/conda/p… /fs/site7/eccc/crd/cccma/users/scrd115/REF/REF_v0.16.2/climate-ref/.ref/cache/pmp-clima…
```
[climate-ref_2026-08-18_22-17_SpawnProcess-1.log](https://github.com/user-attachments/files/31200343/climate-ref_2026-08-18_22-17_SpawnProcess-1.log)
[log.txt](https://github.com/user-attachments/files/31200342/log.txt)
[main_log.txt](https://github.com/user-attachments/files/31200344/main_log.txt)
Contributor guide
Assessment
This issue has not been assessed yet.