FIPS issue submitting DDPJobDefinition job from the CodeFlare Notebook

Describe the Bug

On non-FIPS, when you submit the guided-demos/2_basic_jobs DDPJobDefinition mnisttest, the job is scheduled as pending, then switches to running and then completes.

On a FIPS cluster, I'm noticing the following error - (I'll post the entire output below in a comment)

Issue with path: /tmp/torchx_workspacel83oit3q
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
File /opt/app-root/lib64/python3.8/site-packages/ray/_private/runtime_env/working_dir.py:64, in upload_working_dir_if_needed(runtime_env, scratch_dir, logger, upload_fn)
     63 try:
---> 64     working_dir_uri = get_uri_for_directory(working_dir, excludes=excludes)
     65 except ValueError:  # working_dir is not a directory
....
ValueError: directory /tmp/torchx_workspacel83oit3q must be an existing directory or a zip package

Codeflare Stack Component Versions

Please specify the component versions in which you have encountered this bug.

Codeflare SDK:

pip list |grep codeflare-sdk
codeflare-sdk            0.8.0

MCAD: Unknown, integrated into CodeFlare Operator v1.0.0-rc.1
Instascale: Unknown, integrated into CodeFlare Operator v1.0.0-rc.1
Codeflare Operator: v1.0.0-rc.1
Other: OpenShift 4.12.22 with FIPS enabled:
All master and worker nodes report FIPS enabled, for example:

ssh core@master0.jimfips.cp.fyre.ibm.com cat /proc/sys/crypto/fips_enabled
1
and
ssh core@worker0.jimfips.cp.fyre.ibm.com cat /proc/sys/crypto/fips_enabled
1

Steps to Reproduce the Bug

1. Create a FIPS cluster
2. Install ODH 1.9.0 and CodeFlare v1.0.0-rc1 as usual
3. Install the kfdefs as usual
4. Launch the codeflare notebook as usual
5. Run the guided-demos/2_basic_jobs.ipynb - it works up to where you submit the job, and then reports the Issue with path: /tmp/torchx_workspacel83oit3q issue.

What Have You Already Tried to Debug the Issue?

I tried it on non-FIPS and it worked fine. I also tried a second FIPS cluster to make sure it wasn't just a bad cluster.

Expected Behavior

I expected the job to be scheduled, run and complete successfully.

Screenshots, Console Output, Logs, etc.

More detail of the codeflare-notebook error message will be posted below.

Affected Releases

main

Additional Context

Add as applicable and when known:

• Cloud: 1) AWS, 2) IBM Cloud, 3) Other (describe), or 4) on-premise: [1 - 4 + description?]
• Kubernetes: 1) OpenShift
• OpenShift or K8s version: 4.12.22
• Other relevant info
  Enabled with FIPS

Start by running guided-demos/2_basic_jobs.ipynb with CodeFlare SDK 0.8.0 on an OpenShift 4.12.22 FIPS cluster, then compare the submission with a non-FIPS cluster. Trace the job submission and Ray working-directory handling around the reported /tmp/torchx_workspacel83oit3q path. Done means the DDPJobDefinition is scheduled, runs, and completes successfully on FIPS.