Is it possible to merge two DALIGenericIterator like in torch.utils.data.ConcatDataset?
Open
Nobody has claimed this yet.
enhancement
external contribution welcome
- Dominant language
- C++
- Stars
- 5.8k
- Forks
- 678
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 27
Description
Suppose we have two different paths to two different folders with their jsons.
How can I "merge" them?
What I want to do is to use them as one DALIGenericIterator.
How I would've done it if I've used torch :
- Create 2
torch.utils.data.Datasetobjects - Merge them via
torch.utils.data.ConcatDataset - Create
torch.utils.data.DataLoaderinstance
Any suggestions?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the DALIGenericIterator API and the torch.utils.data.ConcatDataset behavior described in the issue. Determine whether combining two paths and their JSON files should be supported as one iterator, and define the expected iteration and batching behavior before identifying documentation and tests to update.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- data-engineering, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100