awslabs / awslabs/keys_values

Try to use PyTorch DataLoader again

Open
#27 0 comments 0 reactions 0 assignees View on GitHub
enhancement good first issue
Dominant language
Python
Stars
18
Forks
8
PR merge metrics
No merged PRs in 30d

Description

**Is your feature request related to a problem? Please describe.**

We replaced `DataLoader` with a simple `MyDataLoader`, which uses our specific batch sampler as it should be (the PyTorch data loader accesses the batch sampler more than it should). Our batch sampler caters correctly for multiple devices, we do not need clever wrapper code (undocumented).

We'd like to understand why `DataLoader` does not work, and ideally get it back, supporting features like prefetching and memory pinning.

**Describe the solution you'd like**

A data loader which works correctly with our batch sampler in the multi-device case, and is based on the PyTorch implementation.

Contributor guide

Open the contributing guide

Research direction

Start by comparing the current MyDataLoader with the PyTorch DataLoader, focusing on how each accesses the batch sampler in the multi-device case. Done means a PyTorch-based data loader works correctly with the project's batch sampler while supporting prefetching and memory pinning.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.