google / google/grain

Correct way to consume multiple files

Open
#708 1 comment 0 reactions 0 assignees View on GitHub
type:support
Dominant language
Python
Stars
779
Forks
86
Avg merge
2d 6h
Merged PRs (30d)
6

Description

What is the correct way to consume multiple file?

I've implemented a `RandomAccessDataSource` which opens and pre-processes each file.

To me the benefit of grain is that I don't have to manually implement batching or sequences (for RNN), and grain knows how to shape the arrays accordingly. If the solution to consuming multiple files is that I have one `RandomAccessDataSource` per file, wrap those in `DataLoader`s, and then for loop over the `DataLoader`s/files, then I lose the batching benefit - I'm effectively having to implement batching.

Is there a way of telling a `DataLoader` "your data is these N `RandomAccessDataSource`"?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.