Shared Memory issue with multiple instances of Streaming Dataset in a multi-gpu setup

Environment

• OS: [Ubuntu 23.06.30]
• Hardware (GPU, or instance type): [8xV100]

The issue

I am trying Streaming Dataset with Pytorch Lightning. In the setup section of the code I try to initialize multiple instances of Streaming Dataset for test, train and validation. These datasets then are passed the dataloaders. The problem arises with multiprocessing, when these three datasets are initialized concurrently.

  File "/anaconda/envs/testing/lib/python3.9/site-packages/streaming/base/dataset.py", line 373, in __init__
    self._shm_prefix_int, self._locals_shm = get_shm_prefix(my_locals, world)
  File "/anaconda/envs/testing/lib/python3.9/site-packages/streaming/base/shared/prefix.py", line 173, in get_shm_prefix
    shm = SharedMemory(name, True, len(data))
  File "/anaconda/envs/testing/lib/python3.9/site-packages/streaming/base/shared/memory.py", line 41, in __init__
    shm = BuiltinSharedMemory(name, create, size)
  File "/anaconda/envs/testing/lib/python3.9/multiprocessing/shared_memory.py", line 103, in __init__
    self._fd = _posixshmem.shm_open(
FileExistsError: [Errno 17] File exists: '/000009_locals'

This error seems to arise when workers try to create a Shared Memory that has already been created for a previous dataset. The same prefix_int is assigned again while shm creation. When only one dataset is initialized the code works as expected on multiple gpus.
Am I missing something here? If not, then how can I go about creating multiple streaming datasets?

Start with streaming/base/dataset.py around the get_shm_prefix call, then inspect streaming/base/shared/prefix.py and the shared-memory creation shown in the traceback. Reproduce concurrent initialization of train, validation, and test datasets under the multi-GPU setup. Done means multiple Streaming Dataset instances can initialize concurrently without the FileExistsError.