google / google/grain

comparing array_record's ArrayRecordDataSource + grain data loader to huggingface. (Grain is extremely slow)

Open
#688 0 comments 3 reactions 0 assignees View on GitHub
type:performance
Dominant language
Python
Stars
779
Forks
86
Avg merge
2d 6h
Merged PRs (30d)
6

Description

Hello,

I have opened up similar issue in [array_record's github page](https://github.com/google/array_record/issues/143), and in terms of array_record perfomance reading like [dryman](https://github.com/dryman) suggested is right. The problem is how can I use grain as a data loader and sampler, if using `__getitem__` is so slow on which `ArrayRecordDataSource` depends on?. Any ideas what I am doing wrong?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.