Launching sbatch jobs with a conda environment loaded can cause unexpected behavior

Root Cause

• By default, sbatch has --export=ALL, so that all environment variables are exported to the submitted job context.
• conda works by modifying environment variables and then checking them to decide how to behave when commands are run.
• "Ghost" Conda environments can exist when jobs are started from a terminal where a conda environment is loaded. If module load is used, PATH will have unexpected state, causing the Conda base python executable to be used over the Conda environment python executable.

Sequence of Events

• When you submit a job with a Conda environment already loaded, all of the Conda variables are set in the job context.
• In the job context, you then perform a module load Miniforge3 or module load Anaconda3 to make conda available. This puts the /share/apps/rc/software/<module>/bin ahead of /home/<user>/conda/envs/<env>/bin in PATH.
• Then you perform conda activate <env>.
• conda sees that <env> is already loaded using some subset of CONDA_ variables, so it does not modify PATH. This is a "ghost" Conda environment.
• Then which python returns an unexpected environment because the PATH variable is in an unexpected state.

Good Practice for Launching sbatch Jobs

1. Start from a fresh terminal, not a subprocess.
2. If a conda environment is loaded use conda deactivate.
3. Use module reset.
4. Use your required sbatch commands.