amazon-braket / amazon-braket/amazon-braket-containers

Make `braket_container.py` thread safe

Ouverte
#326 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
enhancement
Langage dominant
Python
Étoiles
17
Forks
10
Merge moyen
12 h 33 min
PR mergées (30 j)
11

Description

**Describe the feature you'd like**
The [`braket_container.py`](https://github.com/amazon-braket/amazon-braket-containers/blob/main/src/braket_container.py) script to launch the user-provided algorithm script is not thread safe, currently. When running a multi-node job with parallelization through MPI (the hyperparameter `sagemaker_mpi_enabled` makes SageMaker to invoke the `braket_container.py` with `mpirun`), this can create race conditions in paritcular in the step to download, extract and make available the user-provided code, when running a multi-node job.

The `braket_container.py` script should be made thread safe to account for jobs running on multiple instances or (GPU) cores with `sagemaker_mpi_enabled=True`.

**How would this feature be used? Please describe.**
The user shouldn't have to worry about this feature and, specifically, shouldn't have to change the `braket_container.py` script if they want to use MPI support for the jobs.

**Additional context**
There is an [example](https://github.com/amazon-braket/amazon-braket-examples/blob/dd3828c5db4cdcabae99b6535a630dacf97656c6/examples/nvidia_cuda_q/container/braket_container.py#L316) for a simple workaround for this issue in the amazon-braket-examples repository. I have created [an issue](https://github.com/amazon-braket/amazon-braket-examples/issues/679) there to document this doesn't ultimately solve the problem. But, actually, I think this should be addressed here.

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.