Node Preparation Tasks Unexpectedly Run Multiple/Many Times Per Node

We use HPC Pack for running jobs of python tasks. To ensure the python runtime environment is properly configured on a node before a task launches we use a Node Preparation task to run pipenv (installs python libraries, dependencies, etc).

Our expectation of the NodePrep task is that it will only run once per node for a given job. We have noticed that sometimes a NodePrep task will run many times on the same node over the course of the job. For instance, in a recent job, a node preparation task ran 11 times on the same node. No other jobs that ran concurrently assigned tasks to that node during that time. Under what conditions would/should a NodePrep task run again on the same node? The behavior we observed seems erroneous/unintended.

We have also observed node preparation tasks running even when 100% of the tasks in the job were in a Running state. If a job requires no further resources why would NodePrep be running?

If we can provide any logs to help provide more context or help debug, please let me know.

Thank you.