Climate-REF / Climate-REF/climate-ref

Tests on custom runner are failing regularly

Open
#403 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
26
Forks
14
Avg merge
1d 10h
Merged PRs (30d)
36

Description

The tests using the custom runner are failing regularly: https://github.com/Climate-REF/climate-ref/actions/workflows/ci-integration.yaml. It looks like they are failing on trying to access/modify files that are changed by another process on the same machine. Example errors:
```
FileExistsError: [Errno 17] File exists:
'/home/gha-runner/.cache/climate_ref/v0.7.3/obs4REF/obs4REF/NASA-LaRC/CERES-EBAF
-4-2/mon/rsds/gn/v20230209'
make: *** [Makefile:170: fetch-test-data] Error 1
```
```
FileExistsError: [Errno 17] File exists:
'/home/gha-runner/.cache/climate_ref/v0.7.3/CMIP6/CMIP/NCAR/CESM2/historical/r1i
1p1f1/LImon/snc/gn/v20190308'
make: *** [Makefile:170: fetch-test-data] Error 1
```
and similarly in the ref-sample-data repository:
```
Downloading Pixi
Destination file path /home/gha-runner/.pixi/bin/pixi already exists
Waiting 20 seconds before trying again
Destination file path /home/gha-runner/.pixi/bin/pixi already exists
Waiting 13 seconds before trying again
Error: Destination file path /home/gha-runner/.pixi/bin/pixi already exists
```
```
Error: Unable to locate executable file: /home/gha-runner/.pixi/bin/pixi. Please verify either the file path exists or the file can be found within a directory specified by the PATH environment variable. Also check the file mode to verify the file is executable.
```
Are these sharing the same filesystem @lewisjared? It would be nice if they could all start from a clean container and not share a filesystem, but if this is not possible, it would be good to configure them such that they use unique paths.

Related to that I realized that the commands `ref datasets fetch-data`, `ref datasets fetch-sample-data`, and `ref providers create-env` are not safe if the same user runs them more than once in parallel. Maybe we could make these safe using a [filelock](https://py-filelock.readthedocs.io/en/latest/index.html).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.