comparing array_record's ArrayRecordDataSource + grain data loader to huggingface. (Grain is extremely slow)
Open
type:performance
- Dominant language
- Python
- Stars
- 779
- Forks
- 86
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 6
Description
Hello,
I have opened up similar issue in [array_record's github page](https://github.com/google/array_record/issues/143), and in terms of array_record perfomance reading like [dryman](https://github.com/dryman) suggested is right. The problem is how can I use grain as a data loader and sampler, if using `__getitem__` is so slow on which `ArrayRecordDataSource` depends on?. Any ideas what I am doing wrong?
Contributor guide
Assessment
This issue has not been assessed yet.