Climate-REF / Climate-REF/climate-ref

Enable REF to Use Existing Conda Environments on HPC Systems

Open
#359 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
26
Forks
14
Avg merge
1d 10h
Merged PRs (30d)
36

Description

## The motivation

Bringing in the perspective of modelling centres and users intending to run the REF on HPC systems.
The REF currently manages its own conda environments, which I think is a great approach for future ESGF deployments. However, it would be beneficial to also support the use of existing, pre-managed conda environments.

For example, at ACCESS-NRI we maintain curated conda environments for our modelling community. These are already optimised, regularly updated, and aligned with institutional policies. It would make sense to allow REF to point to these existing environments instead of creating new ones from scratch.

Tagging @bouweandela, @lewisjared

## The proposed solution

Introduce an option to specify a provider-managed conda environment path.
This could look something like:

```bash
ref provider set-env /path/to/conda/env
```

Alternatively, it could be a flag to bypass environment creation altogether and use what is already available on the system.

## Alternatives
@lewisjared, would a symlink in the `software` folder pointing to the conda executable I want to run for a provider be sufficient?

## Additional context

HPC facilities like NCI have policies that limit the proliferation of user-created conda environments. These environments can be quite large and place significant pressure on inode usage, which is closely monitored and sometimes restricted.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.