How to split web dataset shards in parallel external source with worker id in DALI?

Describe the question.

I seek to write external source to load webdatasets from S3. And I want to adopt the parallel external source to accelerate the loading.

Usually, the shards of webdatasets will be split based on both the rank id and the worker id (of multi-processing), to ensure there are no overlapping among the processes (an example implementation torchdata.datapipes.iter.ShardingFilter https://pytorch.org/data/main/generated/torchdata.datapipes.iter.ShardingFilter.html)

So it is necessary to know the worker id to achieve this. In PyTorch, this can be achieved using the API torch.utils.data.get_worker_info https://pytorch.org/docs/stable/data.html#torch.utils.data.get_worker_info .

However, in DALI, there doesn't seem to be a similar API available to obtain the worker's ID. Despite searching through the documentation and issues, I have been unable to find a solution.

Check for duplicates

• I have searched the open bugs/issues and have found no duplicates for this bug report