meta-pytorch / meta-pytorch/data

problem with dataload2 chunk

Open
#1,138 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.3k
Forks
179
Avg merge
6d 1h
Merged PRs (30d)
2

Description

🐛 Describe the bug

Dataloader2 returned DataChunk, and can not be moved to device.

def TFRLoader(path):
    record_pipe = FileLister(path)
    file_pipe = FileOpener(record_pipe, mode="b")
    return file_pipe.load_from_tfrecord().map(tfrecord_praser).batch(batch_size)

rs = MultiProcessingReadingService(num_workers=cfg.num_workers)
train_dataloader = DataLoader2(train_dataset, reading_service=rs)

for i, d in enumerate(train_dataloader):
    d = d.to(device)

AttributeError: 'DataChunk' object has no attribute 'to'

also can not input to model

TypeError: conv2d() received an invalid combination of arguments - got (DataChunk, Parameter, Parameter, tuple, tuple, tuple, int), but expected one of:
 * (Tensor input, Tensor weight, Tensor bias, tuple of ints stride, tuple of ints padding, tuple of ints dilation, int groups)
      didn't match because some of the arguments have invalid types: (DataChunk of [Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor,

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

Currently a work around is convert chunk to tensor

d = torch.stack(d.items)
Versions

Latest

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the TFRLoader and DataLoader2 example, then inspect how DataChunk is produced and how its items differ from the tensor expected by the model. Compare the failing .to(device) and model calls with the torch.stack(d.items) workaround. Done means the dataloader output can be moved to the device and passed to the model without that conversion.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.