llnl / llnl/maestrowf

Ability to pass an environment file when running a study

Open
#474 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
161
Forks
49
PR merge metrics
No merged PRs in 30d

Description

### Desired Enhancement
It'd be nice to have the ability pass an environment file to fill in environment variables rather than modifying the pure spec. Something like:

```bash
maestro run --env-file .env my_spec.yml
```

### Motivation
I'm currently crafting spec files that will be shared with members of my group ideally for reproducibility via a github repo. There are some environment variables that I'm setting in the spec that should be fixed, but others that will be user specific. For example, here's a small example from one of the current specs:

```yaml
env:
variables:
# Fixed Configuration that shouldn't be changed
CALI_CONFIG: "runtime-profile,output=amg_caliper_data.cali"
MPICH_GPU_SUPPORT_ENABLED: 1
OUTPUT_PATH: studies/

# Environment Paths that need to be set based on user
SPACK_ENV: # <-- Path to user's pack environment
PYTHON_ENV: # <-- Path to user's target python environment
```

I know the trivial solution would just be to tell users to only modify that section of the spec. But I'd argue it'd be cleaner to provide a `.env.example` in my repo containing the necessary variables which can be changed to `.env` be passed like the initial example.

```env
SPACK_ENV=/path/to/my/spack/env
PYTHON_ENV=/path/to/my/python/env
```

That way it's clear which variables should and shouldn't be changed by the user, and the .env file can be ignored by git, preventing the likelihood of accidentally pushing a spec with hard-coded user specific paths.

Contributor guide

Open the contributing guide

Research direction

Start at the `maestro run` command entry point and trace how the study spec's environment variables are loaded. Add support for the proposed `--env-file .env` option, preserving fixed variables in the spec while filling user-specific values from the file. Done means a study can run with environment values supplied externally without editing the spec.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
63/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.