amazon-braket / amazon-braket/amazon-braket-containers

Make `braket_container.py` thread safe

Abierto
#326 0 comentarios 0 reacciones 0 asignados Ver en GitHub
enhancement
Lenguaje dominante
Python
Estrellas
17
Forks
10
Merge medio
12 h 33 min
PR fusionados (30 d)
11

Descripción

**Describe the feature you'd like**
The [`braket_container.py`](https://github.com/amazon-braket/amazon-braket-containers/blob/main/src/braket_container.py) script to launch the user-provided algorithm script is not thread safe, currently. When running a multi-node job with parallelization through MPI (the hyperparameter `sagemaker_mpi_enabled` makes SageMaker to invoke the `braket_container.py` with `mpirun`), this can create race conditions in paritcular in the step to download, extract and make available the user-provided code, when running a multi-node job.

The `braket_container.py` script should be made thread safe to account for jobs running on multiple instances or (GPU) cores with `sagemaker_mpi_enabled=True`.

**How would this feature be used? Please describe.**
The user shouldn't have to worry about this feature and, specifically, shouldn't have to change the `braket_container.py` script if they want to use MPI support for the jobs.

**Additional context**
There is an [example](https://github.com/amazon-braket/amazon-braket-examples/blob/dd3828c5db4cdcabae99b6535a630dacf97656c6/examples/nvidia_cuda_q/container/braket_container.py#L316) for a simple workaround for this issue in the amazon-braket-examples repository. I have created [an issue](https://github.com/amazon-braket/amazon-braket-examples/issues/679) there to document this doesn't ultimately solve the problem. But, actually, I think this should be addressed here.

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.