amazon-braket / amazon-braket/amazon-braket-containers
Make `braket_container.py` thread safe
- Lenguaje dominante
- Python
- Estrellas
- 17
- Forks
- 10
- Merge medio
- 12 h 33 min
- PR fusionados (30 d)
- 11
Descripción
**Describe the feature you'd like**
The [`braket_container.py`](https://github.com/amazon-braket/amazon-braket-containers/blob/main/src/braket_container.py) script to launch the user-provided algorithm script is not thread safe, currently. When running a multi-node job with parallelization through MPI (the hyperparameter `sagemaker_mpi_enabled` makes SageMaker to invoke the `braket_container.py` with `mpirun`), this can create race conditions in paritcular in the step to download, extract and make available the user-provided code, when running a multi-node job.
The `braket_container.py` script should be made thread safe to account for jobs running on multiple instances or (GPU) cores with `sagemaker_mpi_enabled=True`.
**How would this feature be used? Please describe.**
The user shouldn't have to worry about this feature and, specifically, shouldn't have to change the `braket_container.py` script if they want to use MPI support for the jobs.
**Additional context**
There is an [example](https://github.com/amazon-braket/amazon-braket-examples/blob/dd3828c5db4cdcabae99b6535a630dacf97656c6/examples/nvidia_cuda_q/container/braket_container.py#L316) for a simple workaround for this issue in the amazon-braket-examples repository. I have created [an issue](https://github.com/amazon-braket/amazon-braket-examples/issues/679) there to document this doesn't ultimately solve the problem. But, actually, I think this should be addressed here.
Guía de contribución
Evaluación
Este issue todavía no se ha evaluado.